System and method for responding to driver state

ABSTRACT

A method for controlling vehicle systems includes receiving monitoring information from one or more monitoring systems and determining a plurality of driver states based on the monitoring information from the one or more monitoring systems. The method includes determining a combined driver state based on the plurality of driver states and modifying control of one or more vehicle systems based on the combined driver state.

RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/385,108 filed on Apr. 16, 2019, and now published as U.S. Pub. No.2019/0241190, which is expressly incorporated herein by reference. U.S.application Ser. No. 16/385,108 is a continuation application of U.S.application Ser. No. 15/720,489 filed on Sep. 29, 2017, and published asU.S. Pub. No. 2018/0022358, and now issued as U.S. Pat. No. 10,308,258,which is also expressly incorporated herein by reference.

U.S. application Ser. No. 15/720,489 is a continuation application ofU.S. application Ser. No. 15/656,595 filed on Jul. 21, 2017, publishedas U.S. Pub. No. 2017/0341658, and now issued as U.S. patent Ser. No.10/246,098 on Apr. 2, 2019, which is expressly incorporated byreference. U.S. application Ser. No. 15/656,595 is a continuationapplication of U.S. application Ser. No. 14/851,753 filed on Sep. 11,2015, published as U.S. Pub. No. 2016/0001781, and now issued as U.S.Pat. No. 9,751,534 on Sep. 5, 2017, which is also expressly incorporatedherein by reference. U.S. application Ser. No. 14/851,753 is acontinuation application of International Application No. PCT/US15/37019filed on Jun. 22, 2015, which is further expressly incorporated hereinby reference.

International Application No. PCT/US15/37019 claims priority to U.S.Prov. Application Ser. No. 62/016,037 filed on Jun. 23, 2014 and U.S.Prov. Application Ser. No. 62/098,565 filed on Dec. 31, 2014, both ofwhich are expressly incorporated herein by reference. In the UnitedStates, International Application No. PCT/US15/37019 is acontinuation-in-part of U.S. application Ser. No. 14/573,778 filed onDec. 17, 2014, published as U.S. Pub. No. 2015/0367858, and now issuedas U.S. Pat. No. 9,352,751 on May 31, 2016, which claims priority toU.S. Prov. Application Ser. No. 62/016,020 filed on Jun. 23, 2014; acontinuation-in-part of U.S. application Ser. No. 14/697,593 filed onApr. 27, 2015, published as U.S. Pub. No. 2015/0229341, and now issuedas U.S. Pat. No. 10,153,796 on Dec. 11, 2018, which is acontinuation-in-part of U.S. application Ser. No. 13/858,038 filed onApr. 6, 2013, published as U.S. Pub. No. 2014/0303899, and now issued asU.S. Pat. No. 9,272,689 on Mar. 1, 2016; a continuation-in-part of U.S.application Ser. No. 14/733,836 filed on Jun. 8, 2015, and issued asU.S. Pat. No. 9,475,521 on Oct. 25, 2016; a continuation-in-part of U.S.application Ser. No. 14/744,247 filed on Jun. 19, 2015, and issued asU.S. Pat. No. 9,475,389 on Oct. 25, 2016; a continuation-in-part of U.S.application Ser. No. 14/315,726 filed on Jun. 26, 2014, published asU.S. Pub. No. 2014/0309881, and issued as U.S. Pat. No. 9,505,402 onNov. 29, 2016; and a continuation-in-part of U.S. application Ser. No.14/461,530 filed on Aug. 18, 2014, published as U.S. Pub. No.2014/0371984, and now issued as U.S. Pat. No. 9,440,646 on Sep. 13,2016; all of the foregoing are expressly incorporated herein byreference.

Further, U.S. application Ser. No. 14/851,753 claims priority to U.S.Prov. Application Ser. No. 62/098,565 filed on Dec. 31, 2014, whichagain is expressly incorporated herein by reference.

Additionally, U.S. application Ser. No. 14/851,753 is acontinuation-in-part of U.S. application Ser. No. 13/843,077 filed onMar. 15, 2013, published as U.S. Pub. No. 2014/0276112, and now issuedas U.S. Pat. No. 9,420,958 on Aug. 23, 2016; a continuation-in-part ofU.S. application Ser. No. 14/074,710 filed on Nov. 7, 2013, published asU.S. Pub. No. 2015/0126818, and now issued as U.S. Pat. No. 9,398,875 onJul. 26, 2016; a continuation-in-part of U.S. application Ser. No.14/573,778 filed on Dec. 17, 2014, published as U.S. Pub. No.2015/0367858, and now issued as U.S. Pat. No. 9,352,751 on May 31, 2016,which claims priority to U.S. Prov. Application Ser. No. 62/016,020filed on Jun. 23, 2014; a continuation-in-part of U.S. application Ser.No. 14/697,593 filed on Apr. 27, 2015, published as U.S. Pub. No.2015/0229341, and now issued as U.S. Pub. No. 10,153,796 on Dec. 11,2018, which is a continuation-in-part of U.S. application Ser. No.13/858,038 filed on Apr. 6, 2013, published as U.S. Pub. No.2014/0303899, and now issued as U.S. Pat. No. 9,272,689 on Mar. 1, 2016;a continuation-in-part of U.S. application Ser. No. 14/733,836 filed onJun. 8, 2015, and now issued as U.S. Pat. No. 9,475,521 on Oct. 25,2016; and a continuation-in-part of U.S. application Ser. No. 14/744,247filed on Jun. 19, 2015, and now issued as U.S. Pat. No. 9,475,389 onOct. 25, 2016; all of the foregoing are expressly incorporated herein byreference.

Additionally, in the United States, International Application No.PCT/US15/37019, and thus this application, expressly incorporates hereinby reference the following: U.S. application Ser. No. 13/030,637 filedon Feb. 18, 2011, published as U.S. Pub. No. 2012/0212353 on Aug. 23,2012, and now issued as U.S. Pat. No. 8,698,639 on Apr. 15, 2014; U.S.application Ser. No. 13/843,194 filed on Mar. 15, 2013, published asU.S. Pub. No. 2013/0226408 on Aug. 29, 2013, and now issued as U.S. Pat.No. 9,292,471 on Mar. 22, 2016; U.S. application Ser. No. 13/843,249filed on Mar. 15, 2013, published as U.S. Pub. No. 2013/0245886 on Sep.19, 2013, and now issued as U.S. Pat. No. 9,296,382 on Mar. 29, 2016;U.S. application Ser. No. 13/195,675 filed on Aug. 1, 2011, published asU.S. Pub. No. 2013/0033382 on Feb. 7, 2013, and now issued as U.S. Pat.No. 8,941,499 on Jan. 27, 2015; U.S. application Ser. No. 13/023,323filed on Feb. 8, 2011, and published as U.S. Pub. No. 2012/0202176 onAug. 9, 2012; U.S. application Ser. No. 13/843,077 filed on Mar. 15,2013, published as U.S. Pub. No. 2014/0276112 on Sep. 18, 2014, and nowissued as U.S. Pat. No. 9,420,958 on Aug. 23, 2016; and U.S. applicationSer. No. 14/074,710 filed on Nov. 7, 2013, published as U.S. Pub. No.2015/0126818 on May 7, 2015, and now issued as U.S. Pat. No. 9,398,875on Jul. 26, 2016; all of the foregoing again are expressly incorporatedherein by reference.

BACKGROUND

The current embodiment relates to motor vehicles and in particular to asystem and method for responding to driver state.

Motor vehicles are operated by drivers in various conditions. Lack ofsleep, monotonous road conditions, use of items, or health-relatedconditions can increase the likelihood that a driver can become drowsyor inattentive while driving. Drowsy or inattentive drivers can havedelayed reaction times.

SUMMARY

In one aspect, a method of controlling vehicle systems in a motorvehicle includes, receiving monitoring information from one or moremonitoring systems, determining a plurality of driver states based onthe monitoring information from the one or more monitoring systems anddetermining a combined driver state index based on the plurality ofdriver states. The method also includes modifying control of one or morevehicle systems based on the combined driver state index.

In another aspect, a method of controlling vehicle systems in a motorvehicle includes, receiving monitoring information from one or moremonitoring systems, determining a first driver state and a second driverstate based on the monitoring information from the one or moremonitoring systems and determining a combined driver state index basedon the first driver state and the second driver state. The method alsoincludes modifying the control of one or more vehicle systems based onthe combined driver state index.

In another aspect, a method of controlling vehicle systems in a motorvehicle includes, receiving monitoring information from one or moremonitoring systems, determining a plurality of driver states based onthe monitoring information from the one or more monitoring systems anddetermining a combined driver state index based on the plurality ofdriver states. The method also includes modifying control of one or morevehicle systems based on the combined driver state index.

In another aspect, a method of controlling vehicle systems in a motorvehicle includes, receiving monitoring information from one or moremonitoring systems, determining a plurality of driver states based onthe monitoring information from the one or more monitoring systems anddetermining a combined driver state index based on the plurality ofdriver states. The method also includes operating one or more vehiclesystem based on the combined driver state index.

In another aspect, a method of controlling vehicle systems in a motorvehicle includes, receiving monitoring information from a plurality ofmonitoring systems, determining a plurality of driver states based onthe monitoring information from the plurality of monitoring systems anddetermining a combined driver state index based on the plurality ofdriver states. The method also includes operating one or more vehiclesystems based on the combined driver state index.

Other systems, methods, features and advantages will be, or will become,apparent to one of ordinary skill in the art upon examination of thefollowing figures and detailed description. It is intended that all suchadditional systems, methods, features and advantages be included withinthis description and this summary, be within the scope of theembodiments, and be protected by the following claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments can be better understood with reference to the followingdrawings and detailed description. The components in the figures are notnecessarily to scale, emphasis instead being placed upon illustratingthe principles of the embodiments. Moreover, in the figures, likereference numerals designate corresponding parts throughout thedifferent views.

FIG. 1A is a schematic view of an embodiment of various components andsystems of a motor vehicle;

FIG. 1B is a block diagram of an embodiment of the ECU of FIG. 1A;

FIG. 2 is a schematic view of an embodiment of various different vehiclesystems;

FIG. 3 is a schematic view of an embodiment of various differentmonitoring systems;

FIG. 4 is a perspective view of an exemplary vehicle seat, includingvarious sensors, and an associated seat belt that may be used toselectively couple an occupant to the seat;

FIG. 5 is a block diagram of an exemplary computing device that may beused with the seat and seat belt shown in FIG. 4;

FIG. 6 is a schematic view of a heart rate monitoring system fordetermining changes in a driver state according to an exemplaryembodiment;

FIG. 7 is a process flow diagram of a method for determining changes ina driver state that can be implemented with the system of FIG. 6according to an exemplary embodiment;

FIG. 8 is a schematic view of locations on an individual for measuringcardiac activity;

FIG. 9A is a schematic representation of a cardiac waveform of anelectrical signal representing cardiac activity;

FIG. 9B is a schematic representation of a series of cardiac waveformsof FIG. 9A;

FIG. 10A is a schematic representation of a cardiac waveform of anacoustic signal representing cardiac activity;

FIG. 10B is a schematic representation of a series of cardiac waveformsof FIG. 10A;

FIG. 10C is a schematic representation of a cardiac waveform of anoptical signal representing cardiac activity;

FIG. 10D is a schematic representation of a series of cardiac waveformsof FIG. 10C;

FIG. 11 is a schematic view of a system for biological signal analysisaccording to an exemplary embodiment;

FIG. 12 is a top schematic view of a multidimensional sensor arrayimplemented in the system of FIG. 11 according to an exemplaryembodiment;

FIG. 13 is an orthographic view of the multidimensional sensor array ofFIG. 12;

FIG. 14 is a schematic view of the system of FIG. 11 implemented in avehicle according to an exemplary embodiment;

FIG. 15 is a schematic electric circuit diagram of the multidimensionalsensor array of FIG. 12;

FIG. 16A is a side view of a motor vehicle according to an exemplaryembodiment;

FIG. 16B is an overhead view of the motor vehicle shown in FIG. 16Aincluding exemplary head looking directions according to an exemplaryembodiment;

FIG. 17 illustrates a head coordinate frame of a driver's head accordingto an exemplary embodiment;

FIG. 18 is an illustrative example of a touch steering wheel accordingto an exemplary embodiment;

FIG. 19 a schematic view of a vehicle having an information transferrate system;

FIG. 20 is a schematic detailed view of an information transfer ratesystem of FIG. 19 for determining an information transfer rate;

FIG. 21 is a process flow diagram of a method for determining aninformation transfer rate between a driver and a vehicle;

FIG. 22 is a schematic view of an illustrative computing environment fora computer system for personal identification in a vehicle according toan exemplary embodiment;

FIG. 23 is a process flow diagram of an exemplary method for identifyinga vehicle occupant that can be implemented with the system of FIG. 22;

FIG. 24A is an embodiment of a process of controlling vehicle systemsaccording to driver state;

FIG. 24B is an embodiment of a process of controlling vehicle systemsaccording to driver state similar to FIG. 24 but includingidentification of a driver;

FIG. 25 is a table showing the impact of a response system on variousvehicle systems;

FIG. 26 is an embodiment of a process of determining a level ofdistractedness and operating one or more vehicle systems;

FIG. 27 is an embodiment of a process for operating a vehicle systemusing a control parameter;

FIG. 28 is an embodiment of a relationship between driver state indexand a control coefficient;

FIG. 29 is an embodiment of a calculation unit for determining a controlparameter;

FIG. 30 is an embodiment of a relationship between driver state indexand a vehicle system status;

FIG. 31 is a schematic view of an embodiment of a method of monitoringautonomic nervous system information to determine driver state;

FIG. 32 is an embodiment of a process of monitoring autonomic nervoussystem information to determine driver state;

FIG. 33 is a schematic view of an embodiment of a method of monitoringthe eye movement of a driver to help determine driver state;

FIG. 34 is an embodiment of a process of monitoring eye movement of adriver to determine driver state;

FIG. 35 is a schematic view of an embodiment of a method of monitoringthe head movement of a driver to determine driver state;

FIG. 36 is an embodiment of a process of monitoring the head movement ofa driver to determine driver state;

FIG. 37 is a schematic view of an embodiment of a method of monitoringthe distance between the driver's head and a headrest to determinedriver state;

FIG. 38 is an embodiment of a process of monitoring the distance betweenthe driver's head and a headrest to determine driver state;

FIG. 39 is a flow chart of a method of an embodiment of a process fordetecting driver state by monitoring hand contact and positioninformation with respect to a steering wheel;

FIG. 40 is a schematic view of an embodiment of a method of monitoringsteering information to determine driver state;

FIG. 41 is an embodiment of a process of monitoring steering informationto determine driver state;

FIG. 42 is a schematic view of an embodiment of a method of monitoringlane departure information to determine driver state;

FIG. 43 is an embodiment of a process of monitoring lane departureinformation to determine driver state;

FIG. 44 is a flow chart of a method of an embodiment of a process forcontrolling one or more vehicle systems in a motor vehicle depending ona combined driver state based on a plurality of driver states accordingto an exemplary embodiment;

FIG. 45 is a flow chart of a method of an embodiment of a process forcontrolling one or more vehicle systems in a motor vehicle depending ona combined driver state based on a plurality of driver state levelsaccording to an exemplary embodiment;

FIG. 46 is a flow chart of a method of an embodiment of a process forcontrolling one or more vehicle systems in a motor vehicle based on oneor more combined driver states according to an exemplary embodiment;

FIG. 47 is a schematic view of how a combined driver state index can beused to retrieve a control coefficient according to an exemplaryembodiment;

FIG. 48 is a schematic diagram illustrating an embodiment of a generalrelationship between the combined driver state index of the driver and asystem status according to an exemplary embodiment;

FIG. 49 is a schematic view of an AND logic gate for combining aplurality of driver states (i.e., two driver states) according to anexemplary embodiment;

FIG. 50 is a schematic view of an AND logic gate for combining aplurality of driver states (i.e., three driver states) according to anexemplary embodiment;

FIG. 51 is a schematic view of an AND/OR logic gate for combining aplurality of driver states (i.e., three driver states) according to anexemplary embodiment;

FIG. 52 is a flow chart of a method of an embodiment of a process forcontrolling one or more vehicle systems in a motor vehicle depending ona combined driver state using thresholds according to an exemplaryembodiment;

FIG. 53 is a schematic view of an AND logic gate for combining aplurality of driver states (i.e., three driver states) with thresholdsaccording to an exemplary embodiment;

FIG. 54 is a flow chart of a method of an embodiment of a process fordetermining and/or modifying a threshold, control parameter, and/orcontrol coefficient according to an exemplary embodiment;

FIG. 55 is a flow chart of a method of an embodiment of a process forcontrolling one or more vehicle systems in a motor vehicle depending ona combined driver state and confirmation of one or more driver states.according to an exemplary embodiment;

FIG. 56 is a flow chart of a method of an embodiment of a process forcontrolling one or more vehicle systems in a motor vehicle depending ona combined driver state and confirmation of one or more driver stateswith thresholds according to an exemplary embodiment;

FIG. 57 is a flow chart of a method of an embodiment of a process forcontrolling one or more vehicle systems in a motor vehicle depending ona combined driver state and confirmation of one or more driver stateswith thresholds according to another exemplary embodiment;

FIG. 58 is a flow chart of a method of an embodiment of a process forcontrolling one or more vehicle systems in a motor vehicle depending ona combined driver state and confirmation of one or more driver states(i.e., three driver states) with thresholds according to anotherexemplary embodiment;

FIG. 59 is a flow chart of a method of an embodiment of a process forconfirming one or more driver states according to a priority level;

FIG. 60 is a network diagram of a multi-modal neural network system forcontrolling one or more vehicle systems according to an exemplaryembodiment;

FIG. 61 is a flow chart of a process of controlling vehicle systemsaccording to a combined driver state index according to anotherexemplary embodiment;

FIG. 62 is a flow chart of a method of an embodiment of a process forcontrolling one or more vehicle systems in a motor vehicle depending onone or more driver states and one or more vehicular states;

FIG. 63 is a schematic view of an embodiment of a method of modifyingthe operation of a power steering system when a driver is drowsy;

FIG. 64 is a schematic view of an embodiment of a method of modifyingthe operation of a power steering system when a driver is drowsy;

FIG. 65 is an embodiment of a process of controlling a power steeringsystem when a driver is drowsy;

FIG. 66 is an embodiment of a detailed process for controlling powersteering assistance in response to driver state;

FIG. 67 is a schematic view of an embodiment of a method of modifyingthe operation of a climate control system when a driver is drowsy;

FIG. 68 is a schematic view of an embodiment of a method of modifyingthe operation of a climate control system when a driver is drowsy;

FIG. 69 is an embodiment of a process of controlling a climate controlsystem when a driver is drowsy;

FIG. 70 is a schematic view of an embodiment of various provisions thatcan be used to wake a drowsy driver;

FIG. 71 is a schematic view of an embodiment of a method of waking up adrowsy driver using tactile devices, visual devices and audio devices;

FIG. 72 is an embodiment of a process for waking up a drowsy driverusing tactile devices, visual devices and audio devices;

FIG. 73 is a schematic view of an electronic pretensioning system for amotor vehicle;

FIG. 74 is a schematic view of a method of waking up a driver using theelectronic pretensioning system of FIG. 73;

FIG. 75 is an embodiment of a process of controlling an electronicpretensioning system according to driver state;

FIG. 76 is a schematic view of an embodiment of a method of operating anantilock braking system when a driver is fully awake;

FIG. 77 is a schematic view of an embodiment of a method of modifyingthe operation of the antilock braking system of FIG. 76 when the driveris drowsy;

FIG. 78 is an embodiment of a process of modifying the operation of anantilock braking system according to driver state;

FIG. 79 is an embodiment of a process of modifying the operation of abrake system according to driver state;

FIG. 80 is an embodiment of a process of modifying the operation of abrake assist system according to driver state;

FIG. 81 is an embodiment of a process for controlling brake assistaccording to driver state;

FIG. 82 is an embodiment of a process for determining an activationcoefficient for brake assist;

FIG. 83 is a schematic view of an embodiment of a motor vehicleoperating with an electronic stability control system;

FIG. 84 is a schematic view of an embodiment of a method of modifyingthe operation of the electronic control assist system of FIG. 83 whenthe driver is drowsy;

FIG. 85 is an embodiment of a process of modifying the operation of anelectronic stability control system according to driver state;

FIG. 86 is an embodiment of a process for controlling an electronicstability control system in response to driver state;

FIG. 87 is an embodiment of a process for setting an activationthreshold for an electronic stability control system;

FIG. 88 is a schematic view of an embodiment of a motor vehicle equippedwith a collision warning system;

FIG. 89 is an embodiment of a process of modifying the control of acollision warning system according to driver state;

FIG. 90 is an embodiment of a detailed process of modifying the controlof a collision warning system according to driver state;

FIG. 91 is a schematic view of an embodiment of a motor vehicleoperating with an automatic cruise control system;

FIG. 92 is a schematic view of an embodiment of a method of modifyingthe control of the automatic cruise control system of FIG. 91 accordingto driver state;

FIG. 93 is an embodiment of a process of modifying the control of anautomatic cruise control system according to driver state;

FIG. 94 is an embodiment of a process of modifying operation of anautomatic cruise control system in response to driver state;

FIG. 95 is an embodiment of a process of modifying a cruising speed of avehicle according to driver state;

FIG. 96 is an embodiment of a process for controlling a low speed followfunction associated with cruise control;

FIG. 97 is a schematic view of an embodiment of a motor vehicleoperating with a lane departure warning system;

FIG. 98 is a schematic view of an embodiment of a method of modifyingthe control of the lane departure warning system of FIG. 97 when thedriver is drowsy;

FIG. 99 is an embodiment of a process of modifying the control of a lanedeparture warning system according to driver state;

FIG. 100 is an embodiment of a process of modifying the operation of alane departure warning system in response to driver state;

FIG. 101 is an embodiment of a process for setting a road crossingthreshold;

FIG. 102 is an embodiment of a process of modifying the operation of alane keep assist system in response to driver state;

FIG. 103 is a schematic view of an embodiment in which a blind spotindicator system is active;

FIG. 104 is a schematic view of an embodiment in which a blind spotindicator system is active and a blind spot monitoring zone is increasedin response to driver state;

FIG. 105 is an embodiment of a process of modifying the control of ablind spot indicator system;

FIG. 106 is an embodiment of a process for controlling a blind spotindicator system is response to driver state;

FIG. 107 is an embodiment of a process for determining a zone thresholdfor a blind spot indicator system;

FIG. 108 is an embodiment of a chart for selecting warning typeaccording to driver state index;

FIG. 109 is a schematic view of an embodiment of a collision mitigationbraking system in which no warning is provided when the driver is alert;

FIG. 110 is a schematic view of an embodiment of a collision mitigationbraking system in which a warning is provided when the driver is drowsy;

FIG. 111 is a schematic view of an embodiment of a collision mitigationbraking system in which no automatic seat belt pretensioning is providedwhen the driver is alert;

FIG. 112 is a schematic view of an embodiment of a collision mitigationbraking system in which automatic seat belt pretensioning is providedwhen the driver is drowsy;

FIG. 113 is an embodiment of a process for controlling a collisionmitigation braking system in response to driver state;

FIG. 114 is an embodiment of a process for setting time to collisionthresholds;

FIG. 115 is an embodiment of a process for operating a collisionmitigation braking system during a first warning stage;

FIG. 116 is an embodiment of a process for operating a collisionmitigation braking system during a second warning stage;

FIG. 117 is an embodiment of a process for operating a navigation systemaccording to driver monitoring;

FIG. 118 is a flow chart of a method of an embodiment of a process formodifying failure thresholds according to an exemplary embodiment;

FIG. 119 is a schematic diagram of an exemplary control signal andfailure detection system thresholds;

FIG. 120 is a flow chart of a method of an embodiment of a process formodifying one or more vehicle systems based on detecting a failure and adriver state according to an exemplary embodiment;

FIG. 121 is a flow chart of a method of an embodiment of a process formodifying failure thresholds according to an exemplary embodiment;

FIG. 122A is a schematic view of modifying a failure threshold accordingto the method of FIG. 121 according to one embodiment;

FIG. 122B is a schematic view of modifying a failure threshold accordingto the method of FIG. 121 according to another embodiment;

FIG. 123 is a schematic view of modifying a failure threshold accordingto the method of FIG. 121;

FIG. 124 is a schematic view of modifying a failure threshold accordingto the method of FIG. 121;

FIG. 125 is a flow chart of an illustrative process of controllingvehicle systems according to combined driver state index using heartrate information and eye movement information according to an exemplaryembodiment;

FIG. 126 is a flow chart of an illustrative process of controllingvehicle systems according to combined driver state index using heartrate information and steering information according to an exemplaryembodiment;

FIG. 127 is a flow chart of a method of an embodiment of a process forcontrolling one or more vehicle systems in a motor vehicle based on acombined driver state and confirmation of one or more driver states withthresholds according to an exemplary embodiment;

FIG. 128 is a flow chart of an illustrative process of controllingvehicle systems according to combined driver state index and a vehicularstate according to an exemplary embodiment;

FIG. 129 is a schematic view of an embodiment of a response systemincluding a central ECU;

FIG. 130 is schematic view of an embodiment of a first vehicle systemand a second vehicle system communicating through a network;

FIG. 131 is an embodiment of a process for modifying the operation ofone or more vehicle systems;

FIG. 132 is an embodiment of a process for controlling selected vehiclesystems in response to driver state;

FIG. 133 is an embodiment of a process for determining a risk levelassociated with a potential hazard;

FIG. 134 is an embodiment of a process for modifying the control of twovehicle systems;

FIG. 135A is a flow chart of a method of an embodiment of a process formodifying control of one or more vehicle systems;

FIG. 135B is a flow chart of a method of an embodiment of a process formodifying control of one or more vehicle systems;

FIG. 136A is a schematic view of an embodiment of a motor vehicleconfigured with a blind spot indicator system;

FIG. 136B is a schematic view of an embodiment of a motor vehicleconfigured with a blind spot indicator system in which the vehicle isswitching lanes;

FIG. 137A is a schematic view of an embodiment of a motor vehicleconfigured with a blind spot indicator system in which the size of ablind spot warning zone is increased as the driver becomes drowsy;

FIG. 137B is a schematic view of an embodiment of a motor vehicleconfigured with a blind spot indicator system and an electronic powersteering system working in cooperation with the blind spot indicatorsystem;

FIG. 138 is an embodiment of a process for controlling a blind spotindicator system in cooperation with an electronic power steeringsystem;

FIG. 139 is a schematic view of an embodiment of a motor vehicleconfigured with a blind spot indicator system with cross-traffic alertand a brake control system working in cooperation with the blind spotindicator system;

FIG. 140 is an embodiment of a process for controlling a blind spotindicator system in cooperation with a brake control system;

FIG. 141 is a flow chart of a method of an embodiment of a process formodifying control of one or more vehicle systems including auto controlaccording to an exemplary embodiment;

FIG. 142 is a flow chart of a method of an embodiment of a process formodifying control of one or more vehicle systems including auto controlaccording to another exemplary embodiment;

FIG. 143A is an exemplary look-up table for auto control of a low speedfollow system based on a driver state according to an exemplaryembodiment;

FIG. 143B is an exemplary look-up table for auto control of a lane keepassist system based on a driver state according to an exemplaryembodiment;

FIG. 143C is an exemplary look-up table for auto control of an automaticcruise control system based on a driver state according to an exemplaryembodiment;

FIG. 143D is an exemplary look-up table for auto control of a visualdevice system based on a driver state according to an exemplaryembodiment;

FIG. 144 is a flow chart of a method of an embodiment of a process forcontrolling one or more vehicle systems including suppressing and/orrestricting vehicle systems according to an exemplary embodiment;

FIG. 145 is a flow chart of a method of an embodiment of a process forcontrolling one or more vehicle systems including confirming a riskand/or hazard according to an exemplary embodiment;

FIG. 146 is a flow chart of a method for an embodiment of controlling alane departure warning system according to an exemplary embodiment;

FIG. 147A is a schematic view of controlling a lane departure warningsystem according to the method of FIG. 146;

FIG. 147B is a schematic view of controlling a lane departure warningsystem according to the method of FIG. 146;

FIG. 148 is a flow chart of a method for an embodiment of controlling ablind spot indicator system according to an exemplary embodiment;

FIG. 149A is a schematic view of controlling a blind spot indicatorsystem according to the method of FIG. 148;

FIG. 149B is a schematic view of controlling a blind spot indicatorsystem according to the method of FIG. 148;

FIG. 150 is a flow chart of a method of an embodiment of a process forcontrolling a lane departure warning system and a blind spot indicatorsystem according to an exemplary embodiment;

FIG. 151A is a schematic view of controlling a lane departure warningsystem and a blind spot indicator system according to the method of FIG.150;

FIG. 151B is a schematic view of controlling a lane departure warningsystem and a blind spot indicator system according to the method of FIG.150;

FIG. 152 is a flow chart of a method of an embodiment of a process forcontrolling an idle mode of an engine according to an exemplaryembodiment;

FIG. 153 is a flow chart of a method of an embodiment of a process forcontrolling a brake hold of an electric parking brake system accordingto an exemplary embodiment;

FIG. 154 is a flow chart of a method of an embodiment of a process forcontrolling an electric parking brake system according to an exemplaryembodiment;

FIG. 155A is a flow chart of a method of an embodiment of a process forcontrolling vehicle systems according to hand contact transitionsaccording to an exemplary embodiment;

FIG. 155B is a flow chart of a method of an embodiment of a process forcontrolling a vehicle mode selector system based in part on hand contacttransitions according to an exemplary embodiment;

FIG. 156 is a flow chart of a method of an embodiment of a process forcontrolling a power steering system according to an exemplaryembodiment;

FIG. 157 is a flow chart of a method of an embodiment of a process forcontrolling a low speed follow system according to an exemplaryembodiment;

FIG. 158A is a schematic view of controlling a low speed follow systemaccording to the method of FIG. 157;

FIG. 158B is a schematic view of controlling a low speed follow systemaccording to the method of FIG. 157;

FIG. 159 is a flow chart of a method of an embodiment of a process forcontrolling a low speed follow system according to another exemplaryembodiment;

FIG. 160 is a flow chart of a method of an embodiment of a process forcontrolling an automatic cruise control system and a lane keep assistsystem according to an exemplary embodiment;

FIG. 161A is a schematic view of controlling an automatic cruise controlsystem and a lane keep assist system according to the method of FIG.160;

FIG. 161B is a schematic view of controlling an automatic cruise controlsystem and a lane keep assist system according to the method of FIG.160;

FIG. 161C is a schematic view of controlling an automatic cruise controlsystem and a lane keep assist system according to the method of FIG.160;

FIG. 162 is a flow chart of a method of an embodiment of a process forcontrolling an automatic cruise control system and a lane keep assistsystem according to another exemplary embodiment;

FIG. 163 is a flow chart of a method of an embodiment of a process forcontrolling an automatic cruise control system and a lane keep assistsystem according to further exemplary embodiment;

FIG. 164 is a flow chart of a method of an embodiment of a process forcontrolling an automatic cruise control system and a lane keep assistsystem according to another exemplary embodiment;

FIG. 165A is a schematic view of controlling an automatic cruise controlsystem and a lane keep assist system according to the method of FIG.164; and

FIG. 165B is a schematic view of controlling an automatic cruise controlsystem and a lane keep assist system according to the method of FIG.164.

DETAILED DESCRIPTION

The following detailed description is intended to be exemplary and thoseof ordinary skill in the art will recognize that other embodiments andimplementations are possible within the scope of the embodimentsdescribed herein. The exemplary embodiments are first describedgenerally with a system overview including the components of a motorvehicle, exemplary vehicle systems and sensors, and monitoring systemsand sensors. After the general description, systems and methods forassessing driver state and operational response including discussions ofdetermining a driver state, determining one or more driver states,determining a combined driver state, and confirming driver states arepresented. Exemplary implementations of detecting the driver states andexemplary operational responses of vehicle systems based on the driverstates and/or combined driver state are also described. Further,embodiments related to various levels of operational response based onthe driver state from no control to semi-autonomous and fully autonomousresponses are also discussed. For organizational purposes, thedescription is structured into sections identified by headings, whichare not intended to be limiting.

I. Overview

The detailed description and exemplary embodiments discussed hereindescribe systems and methods implementing state monitoring of abiological being (e.g., a human, an animal, a driver, a passenger). Inparticular, the detailed description and exemplary embodiments discussedherein refer to methods and systems with respect to a motor vehicle. Forexample, FIG. 1A illustrates a schematic view of an exemplary motorvehicle 100 and various components for implementing systems and methodsfor responding to driver state. In FIG. 1A, the motor vehicle 100 iscarrying a driver 102. In the systems and methods described herein, themotor vehicle 100 and components of the motor vehicle 100 can providestate monitoring of the driver 102 and implement control based on thestate monitoring. The term “driver” as used throughout this detaileddescription and in the claims can refer to any biological being where astate (e.g., a driver state) of the biological being is monitored. Insome situations, the biological being is completing a task that requiresstate monitoring. Examples of the term “driver” can include, but are notlimited to, a driver operating a vehicle, a vehicle occupant, apassenger in a vehicle, a patient, a security guard, an air trafficcontroller, an employee, a student, among others. It is understood thatthese systems and methods can also be implemented outside of a vehicle.Thus, the systems and methods described herein can be implemented in anylocation, situation, or device that requires or implements statemonitoring of a biological being. For example, in any location,situation, or device for monitoring a person executing a task thatrequires a particular state. Examples include, but are not limited to, ahospital location, a home location, a job location, a personal medicaldevice, a portable device, among others.

The “state” of the biological being or “driver state,” as used herein,refers to a measurement of a state of the biological being and/or astate of the environment surrounding (e.g., a vehicle) the biologicalbeing. A driver state or alternatively a “being state” can be one ormore of alert, vigilant, drowsy, inattentive, distracted, stressed,intoxicated, other generally impaired states, other emotional statesand/or general health states, among others. Throughout thisspecification, drowsiness and/or distractedness will be used as theexample driver state being assessed. However, it is understood that anydriver state could be determined and assessed, including but not limitedto, drowsiness, attentiveness, distractedness, vigilance, impairedness,intoxication, stress, emotional states and/or general health states,among others.

A driver state can be quantified as a driver state level, a driver stateindex, among others. Further, one or more driver states can be used todetermine a combined driver state level, a combined driver state index,among others. It is understood that the systems and methods forresponding to driver state discussed herein can include determiningand/or assessing one or more driver states based on information from thesystems and sensors discussed herein. One or more driver states can bebased on various types of information, for example, monitoringinformation, physiological information, behavioral information, vehicleinformation, among others.

As mentioned above, in addition to state monitoring, the systems andmethods described herein can provide one or more responses by the motorvehicle 100 based on driver state. Thus, the assessment and adjustmentdiscussed with the systems and methods herein can accommodate for thedriver's health, slower reaction time, attention lapse and/or alertness.For example, in situations where a driver can be drowsy and/ordistracted, the motor vehicle can include provisions for detecting thatthe driver is drowsy and/or distracted. Moreover, since drowsinessand/or distractedness can increase the likelihood of hazardous drivingsituations, the motor vehicle can include provisions for modifying oneor more vehicle systems automatically to mitigate against hazardousdriving situations. Accordingly, the systems and methods describedherein can monitor and determine a state of a person and provideresponses based on the state (e.g., control the motor vehicle andcomponents of the motor vehicle based on the state). Further, in someembodiments discussed herein, the systems and methods can monitor anddetermine a state of a person and provide automatic control of the motorvehicle and components of the motor vehicle based on driver state.

II. Motor Vehicle Architecture Overview

Referring now to the drawings, wherein the showings are for purposes ofillustrating one or more exemplary embodiments and not for purposes oflimiting the same, an exemplary motor vehicle architecture forresponding to driver state will be described with reference to FIGS. 1Aand 1B. For purposes of clarity, only some components of a motor vehicleare shown in the current embodiment. Furthermore, it will be understoodthat in other embodiments some of the components can be optional. Asmentioned above, FIG. 1A illustrates a schematic view of an exemplarymotor vehicle 100, carrying a driver 102, with various components of themotor vehicle for implementing systems and methods for responding todriver state. The term “motor vehicle” as used throughout this detaileddescription and in the claims refers to any moving vehicle that iscapable of carrying one or more human occupants and is powered by anyform of energy. The term “motor vehicle” includes, but is not limitedto: cars, trucks, vans, minivans, SUVs, motorcycles, scooters, boats,personal watercraft, and aircraft. Further, the term “motor vehicle” canrefer to an autonomous vehicle and/or self-driving vehicle powered byany form of energy. The autonomous vehicle may or may not carry one ormore biological beings (e.g., humans, animals, etc.).

Generally, the motor vehicle 100 can be propelled by any power source.In some embodiments, the motor vehicle 100 can be configured as a hybridvehicle that uses two or more power sources. In other embodiments, themotor vehicle 100 can use one or more engines. For example, in FIG. 1A,the motor vehicle 100 includes a single power source, an engine 104. Thenumber of cylinders in the engine 104 could vary. In some cases, theengine 104 could include six cylinders. In some cases, the engine 104could be a three cylinder, four cylinder, or eight cylinder engine. Instill other cases, the engine 104 could have any other number ofcylinders.

The term “engine” as used throughout the specification and claims refersto any device or machine that is capable of converting energy. In somecases, potential energy is converted to kinetic energy. For example,energy conversion can include a situation where the chemical potentialenergy of a fuel or fuel cell is converted into rotational kineticenergy or where electrical potential energy is converted into rotationalkinetic energy. Engines can also include provisions for convertingkinetic energy into potential energy. For example, some engines includeregenerative braking systems where kinetic energy from a drive train isconverted into potential energy. Engines can also include devices thatconvert solar or nuclear energy into another form of energy. Someexamples of engines include, but are not limited to: internal combustionengines, electric motors, solar energy converters, turbines, nuclearpower plants, and hybrid systems that combine two or more differenttypes of energy conversion processes. It will be understood that inother embodiments, any other arrangements of the components illustratedherein can be used for powering the motor vehicle 100.

Generally, the motor vehicle 100 can include provisions forcommunicating, and in some cases controlling, the various componentsassociated with the engine 104 and/or other systems of the motor vehicle100. In some embodiments, the motor vehicle 100 can include a computeror similar device. In the current embodiment, the motor vehicle 100 caninclude an electronic control unit 106, hereby referred to as the ECU106. In one embodiment, the ECU 106 can be configured to communicatewith, and/or control, various components of the motor vehicle 100.

Referring now to FIG. 1B, an exemplary block diagram of the ECU 106 in aconnected vehicle environment according to one embodiment is shown.Generally, the ECU 106 can include a microprocessor, RAM, ROM, andsoftware all serving to monitor and supervise various parameters of theengine 104, as well as other components or systems of the motor vehicle100. For example, the ECU 106 is capable of receiving signals fromnumerous sensors, devices, and systems located in the engine 104. Theoutput of various devices is sent to the ECU 106 where the devicesignals can be stored in an electronic storage, such as RAM. Bothcurrent and electronically stored signals can be processed by a centralprocessing unit (CPU, processor) in accordance with software stored inan electronic memory, such as ROM.

As illustrated in the embodiment shown in FIG. 1B, the ECU 106 includesa processor 108, a memory 110, a disk 112, and a communication interface114. The processor 108 processes signals and performs general computingand arithmetic functions. Signals processed by the processor can includedigital signals, data signals, computer instructions, processorinstructions, messages, a bit, a bit stream, or other means that can bereceived, transmitted and/or detected. Generally, the processor can be avariety of various processors including multiple single and multicoreprocessors and co-processors and other multiple single and multicoreprocessor and co-processor architectures. The processor, in someembodiments, can include various modules to execute various functions.

The memory 110 can include volatile memory and/or non-volatile memory.Non-volatile memory can include, for example, ROM (read only memory),PROM (programmable read only memory), EPROM (erasable PROM), and EEPROM(electrically erasable PROM). Volatile memory can include, for example,RAM (random access memory), synchronous RAM (SRAM), dynamic RAM (DRAM),synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), and directRAM bus RAM (DRRAM). The memory can store an operating system thatcontrols or allocates resources of the ECU 106.

Further, in some embodiments, the memory 110 can store and facilitateexecution (e.g., by the processor 108) of various software modules 116.The modules, as described herein, can include non-transitory computerreadable medium that stores instructions, instructions in execution on amachine, hardware, firmware, software in execution on a machine, and/orcombinations of each to perform a function(s) or an action(s), and/or tocause a function or action from another module, method, and/or system. Amodule may also include logic, a software controlled microprocessor, adiscrete logic circuit, an analog circuit, a digital circuit, aprogrammed logic device, a memory device containing executinginstructions, logic gates, a combination of gates, and/or other circuitcomponents. Multiple modules may be combined into one module and singlemodules may be distributed among multiple modules. It is understood thatin other embodiments, the software modules 116, could be stored at theprocessor 108 and/or the disk 112.

The disk 112 can be, for example, a magnetic disk drive, a solid statedisk drive, a floppy disk drive, a tape drive, a Zip drive, a flashmemory card, and/or a memory stick. Furthermore, the disk can be aCD-ROM (compact disk ROM), a CD recordable drive (CD-R drive), a CDrewritable drive (CD-RW drive), and/or a digital video ROM drive (DVDROM). The disk can store an operating system that controls or allocatesresources of the ECU 106.

The communication interface 114 provides software and hardware tofacilitate data input and output between the components of the ECU 106and other components, networks and data sources. The processor 108, thememory 110, the disk 112, and the communication interface 114 can eachbe operable connected for computer communication via a data bus 118. Thedata bus 118 refers to an interconnected architecture that is operablyconnected to other computer components inside a computer or betweencomputers. The bus can transfer data between the computer components.The bus can be a memory bus, a memory controller, a peripheral bus, anexternal bus, a crossbar switch, and/or a local bus, among others. Thebus can also be a vehicle bus that interconnects components inside avehicle (e.g., including vehicle systems and sensors) using protocolssuch as Media Oriented Systems Transport (MOST), Controller Area network(CAN), Local Interconnect Network (LIN), among others.

As mention above, the communication interface 114 can facilitate aconnected environment for the motor vehicle 100. Thus, the communicationinterface 114 facilitates the input and output of information to the ECU106, other components of the motor vehicle 100 and other network devicesvia computer communication in a network environment. The computercommunication can include, but is not limited to, a network transfer, afile transfer, a data transfer, an applet transfer, a HTTP transfer, andso on. The computer communication can occur across, for example, logicalconnections, a wireless system (e.g., IEEE 802.11), an Ethernet system(e.g., IEEE 802.3), a token ring system (e.g., IEEE 802.5), a local areanetwork (LAN), a wide area network (WAN), a point-to-point system, acircuit switching system, a packet switching system, among others.

For example, in FIG. 1B, the communication interface 114 can facilitatean operable connection for computer communication to a network 120. Thisconnection can be implemented in various ways, for example, through aportable device 122, a cellular tower 124, a vehicle to vehicle ad-hocnetwork (not shown), an in-vehicle network (not shown), and other wiredand wireless technologies, among others. Accordingly, the motor vehicle100 can transmit data to and receive data from external sources, forexample, the network 120 and the portable device 122.

In addition to the communication interface 114, the ECU 106 can includea number of ports, shown in FIG. 1A, that facilitate the input andoutput of information and power. The term “port” as used throughout thisdetailed description and in the claims refers to any interface or sharedboundary between two conductors. In some cases, ports can facilitate theinsertion and removal of conductors. Examples of these types of portsinclude mechanical connectors. In other cases, ports are interfaces thatgenerally do not provide easy insertion or removal. Examples of thesetypes of ports include soldering or electric traces on circuit boards.In still other cases, ports can facilitate wireless connections.

The ports facilitate the input and output of information to the ECU 106,other components of the motor vehicle 100 and other network devices viacomputer communication in a network environment. The computercommunication can include, but is not limited to, a network transfer, afile transfer, a data transfer, an applet transfer, a HTTP transfer, andso on. The computer communication can occur across, for example, logicalconnections, a wireless system (e.g., IEEE 802.11), an Ethernet system(e.g., IEEE 802.3), a token ring system (e.g., IEEE 802.5), a local areanetwork (LAN), a wide area network (WAN), a point-to-point system, acircuit switching system, a packet switching system, among others. Theports along with the data transfer between the ports and differentvehicle systems will described in more detail herein.

As will be discussed in further detail throughout the detaileddescription, the ports and provisions associated with the ECU 106 areoptional. Some embodiments can include a given port or provision, whileothers can exclude it. The detailed description discloses many of thepossible ports and provisions that can be used, however, it should bekept in mind that not every port or provision must be used or includedin a given embodiment. It is understood that components of the motorvehicle 100 and the ECU 106, as well as the components of other systems,hardware architectures and software architectures discussed herein, maybe combined, omitted or organized into different architecture forvarious embodiments.

III. Systems and Sensors

As mentioned above, one or more driver states can be assessed based onvarious types of information. Different systems and sensors can be usedto gather and/or analyze this information. Generally, sensors discussedherein sense and measure a stimulus (e.g., a signal, a property, ameasurement, a quantity) associated with the motor vehicle 100, avehicle system and/or component, the environment of the motor vehicle100, and/or a biological being (e.g., the driver 102). The sensors cangenerate a data stream and/or a signal representing the stimulus,analyze the signal and/or transmit the signal to another component, forexample the ECU 106. In some embodiments, the sensors are part ofvehicle systems and/or monitoring systems, which will be discussedherein.

The sensors discussed herein can include one sensor, more than onesensor, groups of sensors, and can be part of larger sensing systems,for example, monitoring systems. It is understood that the sensors canbe in various configurations and can include different types of sensors,for example, electric current/potential sensors (e.g., proximity,inductive, capacitive, electrostatic), acoustic sensors, subsonic,sonic, and ultrasonic sensors, vibration sensors (e.g., piezoelectric)visual sensors, imaging sensors, thermal sensors, temperature sensors,pressure sensors, photoelectric sensors, among others.

Exemplary vehicle systems, monitoring systems, sensors and sensoranalysis will now be described in detail. It is understood that thevehicle systems, monitoring systems, sensors, and sensor analysisdescribed herein are exemplary in nature and other systems and sensorscan be implemented with the methods and systems for assessing one ormore driver states and controlling one or more vehicle systems.

A. Vehicle Systems and Sensors

Referring again to FIG. 1A, the motor vehicle 100, the engine 104 and/orthe ECU 106 can facilitate information transfer between components ofthe motor vehicle 100 and/or can facilitate control of the components ofmotor vehicle 100. For example, the components of the motor vehicle 100can include vehicle systems and vehicle sensors. As shown in theembodiments of FIG. 1A and FIG. 2, the motor vehicle 100 can includevarious systems, including vehicle systems 126. The vehicle systems 126can include, but are not limited to, any automatic or manual systemsthat can be used to enhance the vehicle, driving, and/or safety. Themotor vehicle 100 and/or the vehicle systems 126 can include one or morevehicle sensors for sensing and measuring a stimulus (e.g., a signal, aproperty, a measurement, a quantity) associated with the motor vehicle100 and/or a particular vehicle system. In some embodiments, the ECU 106can communicate and obtain data representing the stimulus from thevehicle systems 126 and/or the one or more vehicle sensors via, forexample, a port 128. The data can be vehicle information and/or the ECU106 can process the data into vehicle information and/or process thevehicle information further. Thus, the ECU 106 can communicate andobtain vehicle information from the motor vehicle 100, the vehiclesystems 126 themselves, one or more vehicle sensors associated with thevehicle systems 126, or other vehicle sensors, for example, cameras,external radar and laser sensors, among others.

Vehicle information includes information related to the motor vehicle100 of FIG. 1A and/or the vehicle systems 126, including those vehiclesystems listed in FIG. 2. Specifically, vehicle information can includevehicle and/or vehicle system conditions, states, statuses, behaviors,and information about the external environment of the vehicle (e.g.,other vehicles, pedestrians, objects, road conditions, weatherconditions). Exemplary vehicle information includes, but is not limitedto, acceleration information, velocity information, steeringinformation, lane departure information, blind spot monitoringinformation, braking information, collision warning information,navigation information, collision mitigation information and cruisecontrol information.

It is understood that the vehicle sensors can include, but are notlimited to, vehicle system sensors of the vehicle systems 126 and othervehicle sensors associated with the motor vehicle 100. For example,other vehicle sensors can include cameras mounted to the interior orexterior of the vehicle, radar and laser sensors mounted to the exteriorof the vehicle, external cameras, radar and laser sensors (e.g., onother vehicles in a vehicle-to-vehicle network, street cameras,surveillance cameras). The sensors can be any type of sensor, forexample, acoustic, electric, environmental, optical, imaging, light,pressure, force, thermal, temperature, proximity, among others.

In some embodiments, the ECU 106 can include provisions forcommunicating and/or controlling various systems and/or functionsassociated with the engine 104. In one embodiment, the ECU 106 caninclude a port 130 for receiving various kinds of steering information.In some cases, the ECU 106 can communicate with an electronic powersteering system 132, also referred to as an EPS 132, through the port130. The EPS 132 can comprise various components and devices utilizedfor providing steering assistance. In some cases, for example, the EPS132 can include an assist motor as well as other provisions forproviding steering assistance to a driver. In addition, the EPS 132could be associated with various sensors including torque sensors,steering angle sensors as well as other kinds of sensors. Examples ofelectronic power steering systems are disclosed in Kobayashi, U.S. Pat.No. 7,497,471, filed Feb. 27, 2006 and Kobayashi, U.S. Pat. No.7,497,299, filed Feb. 27, 2006, the entirety of both being herebyincorporated by reference.

In some embodiments, the ECU 106 can include provisions forcommunicating and/or controlling various systems associated with a touchsteering wheel. The ECU 106 can communicate with the various systemsassociated with a touch steering wheel 134 via the port 130 and/or theEPS 132. In the embodiments described herein, the touch steering wheel134 can also be referred to as a touch steering wheel system 134. Thetouch steering wheel system 134 can include various components anddevices utilized for providing information about the contact andlocation of the driver's hands with respect to the touch steering wheel134. More specifically, the touch steering wheel 134 can include sensors(e.g., capacitive sensors, electrodes) mounted in or on the touchsteering wheel 134. The sensors are configured to measure contact of thehands of the driver with the touch steering wheel 134 and a location ofthe contact (e.g., behavioral information). It is understood that insome embodiments, the touch steering wheel 134 can provide contactinformation of other appendages of the driver with the touch steeringwheel 134, for example, wrists, elbows, shoulders, knees, and arms,among others.

In some embodiments, the sensors are located on the front and back ofthe touch steering wheel 134. Accordingly, the sensors can determine ifthe driver's hands are in contact with the front and/or back of thetouch steering wheel 134 (e.g., gripped and wrapped around the steeringwheel). In further embodiments, the touch steering wheel system 134 canmeasure the force and/or pressure of the contact of the hands on thetouch steering wheel 134. In still further embodiments, the touchsteering wheel system 134 can provide information and/or monitormovement of hands on the touch steering wheel 134. For example, thetouch steering wheel system 134 can provide information on a transitionof hand movements or a transition in the number of hands in contact withthe touch steering wheel 134 (e.g., two hands on the touch steeringwheel 134 to one hand on the touch steering wheel 134; one hand on thetouch steering wheel 134 to two hands on the touch steering wheel 134).In some embodiments, a time component can be provided with thetransition in hand contact, for example, a time period between theswitch from two hands on the touch steering wheel 134 to one hand on thetouch steering wheel 134. The information provided by the touch steeringwheel system 134 about contact with the touch steering wheel 134 can bereferred to herein as hand contact information.

In some embodiments, the touch steering wheel system 134 can includesensors to measure a biological parameter of the driver of the vehicle(e.g., physiological information). For example, biological parameterscan include heart rate, skin capacitance, and/or skin temperature. Thesensors can include, for example, one or more bio-monitoring sensors180. In another embodiment, the touch steering wheel 134 can provideinformation for actuating devices and/or functions of vehicle systems.For example, the sensors of the touch steering wheel system 134 canfunction as a switch wherein the contact of the hands of the driver andthe location of the contact are associated with actuating a deviceand/or a vehicle function of the vehicle. In still a further embodiment,the touch steering wheel system 134 can present information to thedriver. For example, the touch steering wheel 134 can include one ormore light elements and/or visual devices to provide information and/orindications to the driver. The light elements and/or visual devices canprovide warning signals and/or information related to one or morevehicle systems. As an illustrative example, the warning signals can beassociated with different visual cues (e.g., colors, patterns). Thevisual cues can be a function of the warning signals and/or driverstate. Examples of touch steering wheel systems are disclosed in U.S.application Ser. No. 14/744,247 filed on Jun. 19, 2015, the entiretybeing hereby incorporated by reference.

In some embodiments, the ECU 106 can include provisions forcommunicating with and/or controlling various visual devices. Visualdevices include any devices that are capable of displaying informationin a visual manner. These devices can include lights (such as dashboardlights, cabin lights, etc.), visual indicators, video screens (such as anavigation screen or touch screen), as well as any other visual devices.In one embodiment, the ECU 106 includes a port 138 for communicatingwith visual devices 140. Further, in one embodiment, the visual devices140 can include light elements and/or visual devices integrated withother vehicle systems, for example the touch steering wheel system 134.

In some embodiments, the ECU 106 can include provisions forcommunicating with and/or controlling various audio devices. Audiodevices include any devices that are capable of providing information inan audible manner. These devices can include speakers as well as any ofthe systems associated with speakers such as radios, DVD players, BDplayers, CD players, cassette players, MP3 players, smartphones,portable devices, navigation systems as well as any other systems thatprovide audio information. In one embodiment, the ECU 106 can include aport 142 for communicating with audio devices 144. Moreover, the audiodevices 144 could be speakers in some cases, while in other cases theaudio devices 144 could include any systems that are capable ofproviding audio information to speakers that can be heard by a driver.

In some embodiments, the ECU 106 can include provisions forcommunicating with and/or controlling various tactile devices. The term“tactile device” as used throughout this detailed description and in theclaims refers to any device that is capable of delivering tactilestimulation to a driver or occupant. For example, a tactile device caninclude any device that vibrates or otherwise moves in a manner that canbe sensed by a driver. Tactile devices could be disposed in any portionof a vehicle. In some cases, a tactile device could be located in asteering wheel (e.g., the touch steering wheel 134) to provide tactilefeedback to a driver. In other cases, a tactile device could be locatedin a vehicle seat (e.g., the vehicle seat 168), to provide tactilefeedback or to help relax a driver. In one embodiment, the ECU 106 caninclude a port 146 for communicating and/or controlling tactile devices148.

In some embodiments, the ECU 106 can include provisions for receivinginput from a user. For example, in some embodiments, the ECU 106 caninclude a port 150 for receiving information from a user input device152. In some cases, the user input device 152 could comprise one or morebuttons, switches, a touch screen, touch pad, dial, pointer or any othertype of input device. For example, in one embodiment, the user inputdevice 152 could be a keyboard or keypad. In another embodiment, theuser input device 152 could be a touch screen. In one embodiment, theuser input device 152 could be an ON/OFF switch. In another embodiment,the user input device 152 can include the touch steering wheel system134. The user input device 152 can receive user input from the touchsteering wheel system 134. In some cases, the user input device 152could be used to turn ON or OFF any driver state monitoring devicesassociated with the vehicle or driver. For example, in an embodimentwhere an optical sensor is used to detect driver state information, theuser input device 152 could be used to switch this type of monitoring ONor OFF. In embodiments using multiple monitoring devices, the user inputdevice 152 can be used to simultaneously turn ON or OFF all thedifferent types of monitoring associated with these monitoring devices.In other embodiments, the user input device 152 can be used toselectively turn ON or OFF some monitoring devices but not others. Infurther embodiments, the user input device 152 can be associated withvehicle systems 126 to selective turn ON or OFF some vehicle systems126.

In some embodiments, the visual devices, audio devices, tactile devicesand/or input devices could be part of a larger infotainment system 154.In FIG. 1A, the ECU can receive information from the infotainment system154 via a port 156. The infotainment system 154 may include a telematicscontrol unit (TCU) (not shown) to allow a connection to the Internet forreceiving various media content. In one embodiment, the TCU canfacilitate connection to a cellular network (e.g., 3G, 4G, LTE). Forexample, the TCU can facilitate connection to the network 120, theportable device 122 and/or the cellular tower 124, similar to thecommunication interface 114. In a further embodiment, the TCU caninclude dedicated short-range communications (DSRC) providing one-way ortwo-way short-range to medium-range wireless communication to thevehicle. Other systems and technologies can be used to allow connectionto the Internet (e.g., network 120) and communicate data between theInternet, other vehicles and other devices. For example, other vehicularcommunication systems (e.g., networks with communication nodes betweenvehicles, other vehicles, roadside units and other devices),vehicle-to-vehicle (V2V) networks allowing communication betweenvehicles, and other ad-hoc networks. It is understood that thecommunication interface 114 shown in FIG. 1B could facilitate thecommunication described above between the infotainment system 154 andother networks and devices.

In some embodiments, the ECU 106 can include ports for communicatingwith and/or controlling various different engine components or systems.Examples of different engine components or systems include, but are notlimited to: fuel injectors, spark plugs, electronically controlledvalves, a throttle, as well as other systems or components utilized forthe operation of the engine 104. Moreover, the ECU 106 could includeadditional ports for communicating with various other systems, sensorsor components of the motor vehicle 100. As an example, in some cases,the ECU 106 could be in electrical communication with various sensorsfor detecting various operating parameters of the motor vehicle 100,including but not limited to: vehicle speed, vehicle acceleration,accelerator pedal input, accelerator pedal input pressure/rate, vehiclelocation, yaw rate, lateral g forces, fuel level, fuel composition,various diagnostic parameters as well as any other vehicle operatingparameters and/or environmental parameters (such as ambient temperature,pressure, elevation, etc.).

In one embodiment, the ECU 106 can include a port 160 for receivinginformation from one or more optical sensing devices, such as an opticalsensing device 162. The optical sensing device 162 could be any kind ofoptical device including a digital camera, video camera, infraredsensor, laser sensor, as well as any other device capable of detectingoptical information. In one embodiment, the optical sensing device 162can be a video camera. In another embodiment, the optical sensing device162 can be one or more cameras or optical tracking systems. In addition,in some cases, the ECU 106 could include a port 164 for communicatingwith a thermal sensing device 166. The thermal sensing device 166 can beconfigured to detect thermal information about the state of a driverand/or thermal information about the vehicle environment. In some cases,the optical sensing device 162 and the thermal sensing device 166 couldbe combined into a single sensor. As will be discussed in further detailherein, the optical sensing device 162 and the thermal sensing device166 can be used to sense and detect physiological and/or behavioralinformation about the driver 102.

As discussed herein, the motor vehicle 100 can include one or moresensors to ascertain, retrieve and/or obtain information about a driver,and more particularly, a driver state. In FIG. 1A, the driver 102 isseated in a vehicle seat 168. The vehicle seat 168 can include a lowersupport 170 and a seat back support 172 that extends generally upwardfrom the lower support 170. Further, the vehicle seat 168 can include aheadrest 174 that extends generally upward from the seat back support172. In some embodiments, the vehicle seat 168 can also include a seatbelt 176. In FIG. 1A, the seat belt 176 is generally shown with a sashbelt portion, however, the seat belt 176 can also include a lap beltportion (not shown). It is understood that other configurations of avehicle seat can be implemented.

The motor vehicle 100 can include one or more bio-monitoring sensors,for example, positioned and/or located in the vehicle seat 168. In FIG.1A, the ECU 106 can include a port 178 for receiving information from abio-monitoring sensor 180 located in the seat back support 172. In afurther embodiment, the ECU 106 can include a port 182 for receivinginformation from a proximity sensor 184 located in the headrest 174. Insome embodiments, the bio-monitoring sensor 180 can be used to sense,receive, and monitor physiological information about the driver 102, forexample, heart rate information. In some embodiments, the proximitysensor 184 can be used to sense, receive and monitor behavioralinformation about the driver 102, for example, a distance between theheadrest 174 and a head 186 of the driver 102. The bio-monitoring sensor180 and the proximity sensor 184 will be described in more detail hereinfor sensing and monitoring physiological and/or behavioral informationabout the driver 102.

In some embodiments, the ECU 106 can include provisions forcommunicating with and/or controlling various other different vehiclesystems. Vehicle systems include any automatic or manual systems thatcan be used to enhance the driving experience and/or enhance safety. Asmentioned above, in one embodiment, the ECU 106 can communicate and/orcontrol vehicle systems 126 via the port 128. For purposes ofillustration, a single port is shown in the current embodiment forcommunicating with the vehicle systems 126. However, it will beunderstood that in some embodiments, more than one port can be used. Forexample, in some cases, a separate port can be used for communicatingwith each separate vehicle system of the vehicle systems 126. Moreover,in embodiments where the ECU 106 comprises part of the vehicle system,the ECU 106 can include additional ports for communicating with and/orcontrolling various different components or devices of a vehicle system.Further, in some embodiments discussed herein, a response system canreceive information about a state of the driver 102 and automaticallyadjust the operation of the vehicle systems 126. In these embodiments,various components, alone or in combination, shown in FIGS. 1A and 1Bcan be referred to herein as a response system 188. In some cases, theresponse system 188 comprises the ECU 106 as well as one or moresensors, components, devices or systems discussed herein.

Examples of different vehicle systems 126 are illustrated in FIG. 2.FIG. 2, also includes the vehicle systems described above in relationwith FIG. 1A, in particular, the EPS 132, the touch steering wheelsystem 134, visual devices 140, tactile devices 148, user input devices152, and infotainment system 154. It should be understood that thesystems shown in FIG. 2 are only intended to be exemplary and in somecases, some other additional systems can be included. In other cases,some of the systems can be optional and not included in all embodiments.FIG. 2 will be described with reference to the components of FIGS. 1Aand 1B. Referring now to FIG. 2, the motor vehicle 100 can include anelectronic stability control system 202 (also referred to as ESC system202). The ESC system 202 can include provisions for maintaining thestability of the motor vehicle 100. In some cases, the ESC system 202can monitor the yaw rate and/or lateral g acceleration of the motorvehicle 100 to help improve traction and stability. The ESC system 202can actuate one or more brakes automatically to help improve traction.An example of an electronic stability control system is disclosed inEllis et al., U.S. Pat. No. 8,423,257, filed Mar. 17, 2010, the entiretyof which is hereby incorporated by reference. In one embodiment, theelectronic stability control system can be a vehicle stability system.

In some embodiments, the motor vehicle 100 can include an antilock brakesystem 204 (also referred to as an ABS system 204). The ABS system 204can include various different components such as a speed sensor, a pumpfor applying pressure to the brake lines, valves for removing pressurefrom the brake lines, and a controller. In some cases, a dedicated ABScontroller can be used. In other cases, ECU 106 can function as an ABScontroller. In still other cases, the ABS system 204 can provide brakinginformation, for example brake pedal input and/or brake pedal inputpressure/rate, among others. Examples of antilock braking systems areknown in the art. One example is disclosed in Ingaki, et al., U.S. Pat.No. 6,908,161, filed Nov. 18, 2003, the entirety of which is herebyincorporated by reference. Using the ABS system 204 can help improvetraction in the motor vehicle 100 by preventing the wheels from lockingup during braking.

The motor vehicle 100 can include a brake assist system 206. The brakeassist system 206 can be any system that helps to reduce the forcerequired by a driver to depress a brake pedal. In some cases, the brakeassist system 206 can be activated for older drivers or any otherdrivers who can need assistance with braking. An example of a brakeassist system can be found in Wakabayashi et al., U.S. Pat. No.6,309,029, filed Nov. 17, 1999, the entirety of which is herebyincorporated by reference.

In some embodiments, the motor vehicle 100 can include an automaticbrake prefill system 208 (also referred to as an ABP system 208). TheABP system 208 includes provisions for prefilling one or more brakelines with brake fluid prior to a collision. This can help increase thereaction time of the braking system as the driver depresses the brakepedal. Examples of automatic brake prefill systems are known in the art.One example is disclosed in Bitz, U.S. Pat. No. 7,806,486, filed May 24,2007, the entirety of which is hereby incorporated by reference.

In some embodiments, the motor vehicle 100 can include an electricparking brake (EPB) system 210. The EPB system 210 includes provisionsfor holding the motor vehicle 100 stationary on grades and flat roads.In particular, the motor vehicle 100 can include an electric park brakeswitch (e.g., a button) that can be activated by the driver 102. Whenactivated, the EPB system 210 controls the braking systems discussedabove to apply braking to one or more wheels of the motor vehicle 100.To release the braking, the driver can engage the electric park brakeswitch and/or press on the accelerator pedal. Additionally, the EPBsystem 210 or other braking systems can include an automatic brake holdcontrol feature that maintains brake hold when the vehicle is stopped,even after the brake pedal is released. Thus, when the vehicle comes toa full stop, brake hold is engaged and the brakes continue to hold untilthe accelerator pedal is engaged. In some embodiments, the automaticbrake hold control feature can be manually engaged with a switch. Inother embodiments, the automatic brake hold control feature is engagedautomatically.

As mentioned above, the motor vehicle 100 includes provisions forcommunicating and/or controlling various systems and/or functionsassociated with the engine 104. In one embodiment, the engine 104includes an idle stop function that can be controlled by the ECU 106and/or the engine 104 based information from, for example, the engine104 (e.g., automatic transmission), the antilock brake system 204, thebrake assist system 205, the automatic brake prefill system 208, and/orthe EPB system 210. Specifically, the idle stop function includesprovisions to automatically stop and restart the engine 104 to helpmaximize fuel economy depending on environmental and vehicle conditions.For example, the ECU 106 can activate the idle stop function based ongear information from the engine 104 (e.g., automatic transmission) andbrake pedal position information from the braking systems describedabove. Thus, when the vehicle stops with a gear position in Drive (D)and the brake pedal is pressed, the ECU 106 controls the engine to turnOFF. When the brake pedal is subsequently released, the ECU 106 controlsthe engine to restart (e.g., turn ON) and the vehicle can begin to move.In some embodiments, when the idle stop function is activated, the ECU106 can control the visual devices 140 to provide an idle stop indicatorto the driver. For example, a visual device 140 on a dashboard of themotor vehicle 100 can be controlled to display an idle stop indicator.Activation of the idle stop function can be disabled in certainsituations based on other vehicle conditions (e.g., seat belt isfastened, vehicle is stopped on a steep hill). Further, the idle stopfunction can be manually controlled by the driver 102 using, forexample, an idle stop switch located in the motor vehicle 100.

In some embodiments, the motor vehicle 100 can include a low speedfollow system 212 (also referred to as an LSF system 212). The LSFsystem 212 includes provisions for automatically following a precedingvehicle at a set distance or range of distances. This can reduce theneed for the driver to constantly press and depress the accelerationpedal in slow traffic situations. The LSF system 212 can includecomponents for monitoring the relative position of a preceding vehicle(for example, using remote sensing devices such as lidar or radar). Insome cases, the LSF system 212 can include provisions for communicatingwith any preceding vehicles for determining the GPS positions and/orspeeds of the vehicles. Examples of low speed follow systems are knownin the art. One example is disclosed in Arai, U.S. Pat. No. 7,337,056,filed Mar. 23, 2005, the entirety of which is hereby incorporated byreference. Another example is disclosed in Higashimata et al., U.S. Pat.No. 6,292,737, filed May 19, 2000, the entirety of which is herebydisclosed by reference.

The motor vehicle 100 can include a cruise control system 214. Cruisecontrol systems are well known in the art and allow a user to set acruising speed that is automatically maintained by a vehicle controlsystem. For example, while traveling on a highway, a driver can set thecruising speed to 55 mph. The cruise control system 214 can maintain thevehicle speed at approximately 55 mph automatically, until the driverdepresses the brake pedal or otherwise deactivates the cruisingfunction.

The motor vehicle 100 can include an automatic cruise control system 216(also referred to as an ACC system 216). In some cases, the ACC system216 can include provisions for automatically controlling the vehicle tomaintain a predetermined following distance behind a preceding vehicleor to prevent a vehicle from getting closer than a predetermineddistance to a preceding vehicle. The ACC system 216 can includecomponents for monitoring the relative position of a preceding vehicle(for example, using remote sensing devices such as lidar or radar). Insome cases, the ACC system 216 can include provisions for communicatingwith any preceding vehicles for determining the GPS positions and/orspeeds of the vehicles. An example of an automatic cruise control systemis disclosed in Arai et al., U.S. Pat. No. 7,280,903, filed Aug. 31,2005, the entirety of which is hereby incorporated by reference.

The motor vehicle 100 can include a collision warning system 218. Insome cases, the collision warning system 218 can include provisions forwarning a driver of any potential collision threats with one or morevehicles, objects and/or pedestrians. For example, a collision warningsystem can warn a driver when another vehicle is passing through anintersection as the motor vehicle 100 approaches the same intersection.Examples of collision warning systems are disclosed in Mochizuki, U.S.Pat. No. 8,558,718, filed Sep. 20, 2010, and Mochizuki et al., U.S. Pat.No. 8,587,418, filed Jul. 28, 2010, the entirety of both being herebyincorporated by reference. In one embodiment, the collision warningsystem 218 could be a forward collision warning system, includingwarning of vehicles and/or pedestrians. In another embodiment, thecollision warning system 218 could be a cross traffic monitoring system,utilizing backup cameras or back sensors to determine if a pedestrian oranother vehicle is behind the vehicle.

The motor vehicle 100 can include a collision mitigation braking system220 (also referred to as a CMBS 220). The CMBS 220 can includeprovisions for monitoring vehicle operating conditions (including targetvehicles, objects, pedestrians in the environment of the vehicle) andautomatically applying various stages of warning and/or control tomitigate collisions. For example, in some cases, the CMBS 220 canmonitor forward vehicles using a radar or other type of remote sensingdevice. If the motor vehicle 100 gets too close to a forward vehicle,the CMBS 220 could enter a first warning stage. During the first warningstage, a visual and/or audible warning can be provided to warn thedriver. If the motor vehicle 100 continues to get closer to the forwardvehicle, the CMBS 220 could enter a second warning stage. During thesecond warning stage, the CMBS 220 could apply automatic seat beltpretensioning. In some cases, visual and/or audible warnings couldcontinue throughout the second warning stage. Moreover, in some cases,during the second stage automatic braking could also be activated tohelp reduce the vehicle speed. In some cases, a third stage of operationfor the CMBS 220 can involve braking the vehicle and tightening a seatbelt automatically in situations where a collision is very likely. Anexample of such a system is disclosed in Bond, et al., U.S. Pat. No.6,607,255, and filed Jan. 17, 2002, the entirety of which is herebyincorporated by reference. The term collision mitigation braking systemas used throughout this detailed description and in the claims can referto any system that is capable of sensing potential collision threats andproviding various types of warning responses as well as automatedbraking in response to potential collisions.

The motor vehicle 100 can include a lane departure warning system 222(also referred to as an LDW system 222). The LDW system 222 candetermine when a driver is deviating from a lane and provide a warningsignal to alert the driver. Examples of lane departure warning systemscan be found in Tanida et al., U.S. Pat. No. 8,063,754, filed Dec. 17,2007, the entirety of which is hereby incorporated by reference.

The motor vehicle 100 can include a blind spot indicator system 224(also referred to as a BSI system 224). The blind spot indicator system224 can include provisions for helping to monitor the blind spot of adriver. In some cases, the blind spot indicator system 224 can includeprovisions to warn a driver if a vehicle is located within a blind spot.In other cases, the blind spot indicator system 224 can includeprovisions to warn a driver if a pedestrian or other object is locatedwithin a blind spot. Any known systems for detecting objects travelingaround a vehicle can be used.

In some embodiments, the motor vehicle 100 can include a lane keepassist system 226 (also referred to as an LKAS system 226). The lanekeep assist system 226 can include provisions for helping a driver tostay in the current lane. In some cases, the lane keep assist system 226can warn a driver if the motor vehicle 100 is unintentionally driftinginto another lane. Also, in some cases, the lane keep assist system 226can provide assisting control to maintain a vehicle in a predeterminedlane. For example, the lane keep assist system 226 can control theelectronic power steering system 132 by applying an amount ofcounter-steering force to keep the vehicle in the predetermined lane. Inanother embodiment, the lane keep assist system 226, in, for example, anautomatic control mode, can automatically control the electronic powersteering system 132 to keep the vehicle in the predetermined lane basedon identifying and monitoring lane markers of the predetermined lane. Anexample of a lane keep assist system is disclosed in Nishikawa et al.,U.S. Pat. No. 6,092,619, filed May 7, 1997, the entirety of which ishereby incorporated by reference.

In some embodiments, the motor vehicle 100 can include a lane monitoringsystem 228. In some embodiments, the lane monitoring system 228 could becombined or integrated with the blind spot indicator system 224 and/orthe lane keep assist system 226. The lane monitoring system 228 includesprovisions for monitoring and detecting the state of the vehicle, andelements in the environment of the vehicle, for example, pedestrians,objects, other vehicles, cross traffic, among others. Upon detection ofsaid elements, the lane monitoring system 228 can warn a driver and/orwork in conjunction with the lane keep assist system 226 to assist inmaintaining control of the vehicle to avoid potential collisions and/ordangerous situations. The lane keep assist system 226 and/or the lanemonitoring system 228 can include sensors and/or optical devices (e.g.,cameras) located in various areas of the vehicle (e.g., front, rear,sides, roof). These sensors and/or optical devices provide a broaderview of the roadway and/or environment of the vehicle. In someembodiments, the lane monitoring system 228 can capture images of a rearregion of a vehicle and a blind spot region of the vehicle out ofviewing range of a side mirror adjacent to the rear region of thevehicle, compress said images and display said images to the driver. Anexample of a lane monitoring system is disclosed in Nishiguichi et al.,U.S. Publication Number 2013/0038735, filed on Feb. 16, 2011, theentirety of which is incorporated by reference. It is understood thatafter detecting the state of the vehicle, the lane monitoring system 228can provide warnings or driver assistances with other vehicles systems,for example, the electronic stability control system 202, the brakeassist system 206, the collision warning system 218, the collisionmitigation braking system 220, the blind spot indicator system 224,among others.

In some embodiments, the motor vehicle 100 could include a navigationsystem 230. The navigation system 230 could be any system capable ofreceiving, sending and/or processing navigation information. The term“navigation information” refers to any information that can be used toassist in determining a location or providing directions to a location.Some examples of navigation information include street addresses, streetnames, street or address numbers, apartment or suite numbers,intersection information, points of interest, parks, any political orgeographical subdivision including town, township, province, prefecture,city, state, district, ZIP or postal code, and country. Navigationinformation can also include commercial information including businessand restaurant names, commercial districts, shopping centers, andparking facilities. In some cases, the navigation system could beintegrated into the motor vehicle, for example, as a part of theinfotainment system 154. Navigation information could also includetraffic patterns, characteristics of roads, and other information aboutroads the motor vehicle currently is travelling on or will travel on inaccordance with a current route. In other cases, the navigation systemcould be a portable, stand-alone navigation system, or could be part ofa portable device, for example, the portable device 122.

As mentioned above, in some embodiments, the visual devices 140, theaudio devices 144, the tactile devices 148 and/or the user input devices152 can be part of a larger infotainment system 154. In a furtherembodiment, the infotainment system 154 can facilitate mobile phoneand/or portable device connectivity to the vehicle to allow, forexample, the playing of content from the mobile device to theinfotainment system. Accordingly, in one embodiment, the vehicle caninclude a hands free portable device (e.g., telephone) system 232. Thehands free portable device system 232 can include a telephone device,for example integrated with the infotainment system, a microphone (e.g.,audio device) mounted in the vehicle. In one embodiment, the hands freeportable device system 232 can include the portable device 122 (e.g., amobile phone, a smart phone, a tablet with phone capabilities). Thetelephone device is configured to use the portable device, themicrophone and the vehicle audio system to provide an in-vehicletelephone feature and/or provide content from the portable device in thevehicle. In some embodiments, the telephone device is omitted as theportable device can provide telephone functions. This allows the vehicleoccupant to realize functions of the portable device through theinfotainment system without physical interaction with the portabledevice.

The motor vehicle 100 can include a climate control system 234. Theclimate control system 234 can be any type of system used forcontrolling the temperature or other ambient conditions in the motorvehicle 100. In some cases, the climate control system 234 can comprisea heating, ventilation and air conditioning system as well as anelectronic controller for operating the HVAC system. In someembodiments, the climate control system 234 can include a separatededicated controller. In other embodiments, the ECU 106 can function asa controller for the climate control system 234. Any kind of climatecontrol system known in the art can be used.

The motor vehicle 100 can include an electronic pretensioning system 236(also referred to as an EPT system 236). The EPT system 236 can be usedwith a seat belt (e.g., the seat belt 176) for the motor vehicle 100.The EPT system 236 can include provisions for automatically tightening,or tensioning, the seat belt 176. In some cases, the EPT system 236 canautomatically pretension the seat belt 176 prior to a collision. Anexample of an electronic pretensioning system is disclosed in Masuda etal., U.S. Pat. No. 6,164,700, filed Apr. 20, 1999, the entirety of whichis hereby incorporated by reference.

The motor vehicle 100 can include a vehicle mode selector system 238that modifies driving performance according to preset parameters relatedto the mode selected. Modes can include, but are not limited to, normal,economy, sport, sport+ (plus), auto, and terrain/condition specificmodes (e.g., snow, mud, off-road, steep grades). For example, in aneconomy mode, the ECU 106 can control the engine 104 (or vehicle systemsrelated to the engine 104) to provide a more consistent engine speedthereby increasing fuel economy. The ECU 106 can also control othervehicle systems to ease the load on the engine 104, for example,modifying the climate control system 234. In a sport mode, the ECU 106can control the EPS 132 and/or the ESC system 202 to increase steeringfeel and feedback. In terrain/condition specific modes (e.g., snow, mud,sand, off-road, steep grades), the ECU 106 can control various vehiclesystems to provide handling, and safety features conducive to thespecific terrain and conditions. In an auto mode, the ECU 106 cancontrol various vehicle systems to provide full (e.g., autonomous) orpartial automatic control of the vehicle. It is understood that themodes and features of the modes described above are exemplary in natureand that other modes and features can be implemented. Further it isappreciated that more than one mode could be implemented at the same orsubstantially the same time.

The motor vehicle 100 can include a turn signal control system 240 forcontrolling turn signals (e.g., directional indicators) and brakingsignals. For example, the turn signal control system 240 can controlturn signal indicator lamps (e.g., mounted on the left and right frontand rear corners of the vehicle, the side of the vehicle, the exteriorside mirrors). The turn signal control system 240 can control (e.g.,turn ON/OFF) the turn signal indicator lamps upon receiving a turnsignal input from the driver (e.g., input via a user input device 152, aturn signal actuator, etc.). In other embodiments, the turn signalcontrol system 240 can control a feature and/or a visual cue of the turnsignal indicator lamps. For example, a brightness, a color, a lightpattern, a mode among others. The feature and/or visual cue control canbe based on input received from the driver or can be an automaticcontrol based on input from another vehicle system and/or a driverstate. For example, the turn signal control system 240 can control theturn signal indicator lamps based on an emergency event (e.g., receivinga signal from the collision warning system) to provide warnings to othervehicles and/or provide information about occupants in the vehicle.Further, the turn signal control system 240 can control braking signals(e.g., braking indicator lamps mounted on the rear of the vehicle) aloneor in conjunction with a braking system discussed herein. The turnsignal control system 240 can also control a feature and/or visual cueof the braking signals similar to the turn signal indicator lampsdescribed above.

The motor vehicle 100 can include a headlight control system 242 forcontrolling headlamps and/or flood lamps mounted on the vehicle (e.g.,located the right and left front corners of the vehicle). The headlightcontrol system 242 can control (e.g., turn ON/OFF, adjust) the headlampsupon receiving an input from the driver. In other embodiments, theheadlight control system 242 can control (e.g., turn ON/OFF, adjust) theheadlamps automatically and dynamically based on information from one ormore of the vehicle systems. For example, the headlight control system242 can actuate the headlamps and/or adjust features of the headlightsbased on environmental/road conditions (e.g., luminance outside,weather), time of day, among others. It is understood that the turnsignal control system 240 and the headlight control system 242 could bepart of a larger vehicle lighting control system.

The motor vehicle 100 can include a failure detection system 244 thatdetects a failure in one or more of the vehicle systems 126. Morespecifically, the failure detection system 244 receives information froma vehicle system and executes a fail-safe function (e.g., system shutdown) or a non-fail-safe function (e.g., system control) based on theinformation and a level of failure. In operation, the failure detectionsystem 244 monitors and/or receives signals from one or more vehiclesystems 126. The signals are analyzed and compared to pre-determinedfailure and control levels associated with the vehicle system. Once thefailure detection system 244 detects the signals meets a pre-determinedlevel, the failure detection system 244 initiates control of the one ormore vehicle systems and/or shuts down the one or more vehicle systems.It is understood that one or more of the vehicle systems 126 couldimplement an independent failure detection system. In some embodiments,the failure detection system 244 can be integrated with an on-boarddiagnostic system of the motor vehicle 100. Further, in someembodiments, the failure detection system 244 could determine failure ofa vehicle system based on a comparison of information from more than onevehicle system. For example, the failure detection system 244 cancompare information indicating hand and/or appendage contact from thetouch steering wheel system 134 and the electronic power steering system132 to determine failure of a touch sensor as described in U.S.application Ser. No. 14/733,836 filed on Jun. 8, 2015 and incorporatedherein by reference.

It is understood that, the vehicle systems 126 could incorporate anyother kinds of devices, components, or systems used with vehicles.Further, each of these vehicle systems can be standalone systems or canbe integrated with the ECU 106. For example, in some cases, the ECU 106can operate as a controller for various components of one or morevehicle systems. In other cases, some systems can comprise separatededicated controllers that communicate with the ECU 106 through one ormore ports.

Further, it is understood that the vehicle systems 126 other vehiclesystems, sensors and monitoring systems discussed herein, for example,the physiological monitoring systems discussed in Section III (B) (1),the behavioral monitoring systems discussed in Section III (B) (2), thevehicular monitoring systems discussed in Section III (B) (3), and theidentification systems and sensors discussed in Section III (B) (4) canbe a vehicle system and/or include vehicle systems and discussed herein.Further, it is appreciated, that any combination of vehicle systems andsensors, physiological monitoring systems, behavioral monitoringsystems, vehicular monitoring systems, and identification systems can beimplemented to determine and/or assess one or more driver statesdiscussed herein.

B. Monitoring Systems and Sensors

Generally, monitoring systems, as used herein, can include any systemconfigured to provide monitoring information related to the motorvehicle 100, the driver 102 of the motor vehicle 100, and/or the vehiclesystems 126. More particularly, these monitoring systems ascertain,retrieve and/or obtain information about a driver, for example,information about a driver state or information to assess a driverstate. In some embodiments, the ECU 106 can communicate and obtainmonitoring information from the monitoring systems and/or one or moremonitoring system sensors, for example, via one or more ports.

Monitoring systems can include, but are not limited to, optical devices,thermal devices, autonomic monitoring devices as well as any other kindsof devices, sensors or systems. More specifically, monitoring systemscan include vehicular monitoring systems, physiological monitoringsystems, behavioral monitoring systems, related sensors, among othersystems and sensors. Further, monitoring information can includephysiological information, behavioral information, and vehicleinformation, among others.

It will be understood that in certain embodiments, vehicle systems andmonitoring systems can be used alone or in combination for receivingmonitoring information. In some cases, monitoring information could bereceived directly from a vehicle system, rather than from a system orcomponent designed for monitoring a driver state. In some cases,monitoring information could be received from both a monitoring systemand a vehicle system. Accordingly, one or more monitoring systems caninclude one or more vehicle systems (FIGS. 1A, 1B 2) and/or one or moremonitoring systems (FIG. 3). Additionally, as mentioned above, and aswill be described in detail below, other additional vehicle systemsand/or monitoring systems can be included that are not shown in FIGS.1A, 1B, 2 and 3.

It will be understood that each of the monitoring systems discussedherein could be associated with one or more sensors or other devices. Insome cases, the sensors could be disposed in one or more portions of themotor vehicle 100. For example, the sensors could be integrated into adashboard, seat (e.g., the seat 168), seat belt (e.g., the seat belt176), door, dashboard, steering wheel (e.g., the touch steering wheelsystem 134), center console, roof or any other portion of the motorvehicle 100. In other cases, however, the sensors could be portablesensors worn by a driver, integrated into a portable device (e.g., theportable device 122) carried by the driver, integrated into an articleof clothing worn by the driver or integrated into the body of the driver(e.g. an implant). Specific types of sensors and sensor placement willbe discussed in more detail below.

Exemplary monitoring systems as well as other exemplary sensors, sensingdevices and sensor analysis (e.g., analysis and processing of datameasured by the sensors) are described in detail below. It isappreciated that one or more components/functions of each of the systemsand methods discussed herein can be implemented within or in conjunctionwith the motor vehicle 100, the components of the motor vehicle 100, thevehicle systems 126, the monitoring systems of FIGS. 1A, 1B, 2 and 3 andthe systems and methods described in relation to FIGS. 1A, 1B, 2, and 3.The exemplary monitoring systems, the sensors, the sensing devices andthe sensor analysis described below generally detect and providemonitoring information and can determine one or more driver states ofthe driver of the motor vehicle 100. The one or more driver states canbe utilized by the methods and systems described in relation to theother figures herein to control and/or modify one or more vehiclesystems. The exemplary monitoring systems, sensors, sensing devices andsensors analysis are non-limiting and components and/or functions of theconfigurations and methods can be reorganized and/or omitted for otherexemplary embodiments, including those related to FIGS. 1A, 1B, 2 and 3.

1. Physiological Monitoring Systems and Sensors

Generally, physiological monitoring systems and sensors include, but arenot limited to, any automatic or manual systems and sensors that monitorand provide physiological information related to a driver of a motorvehicle (e.g., related to a driver state). The physiological monitoringsystems can include one or more physiological sensors for sensing andmeasuring a stimulus (e.g., a signal, a property, a measurement, and/ora quantity) associated with the driver of the motor vehicle 100. In someembodiments, the ECU 106 can communicate and obtain a data streamrepresenting the stimulus from the physiological monitoring system from,for example, a port. In other words, the ECU 106 can communicate andobtain physiological information from the physiological monitoringsystems of the motor vehicle 100.

Physiological information includes information about the human body(e.g., a driver) derived intrinsically. Said differently, physiologicalinformation can be measured by medical means and quantifies an internalcharacteristic of a human body. Physiological information is typicallynot externally observable to the human eye. However, in some cases,physiological information is observable by optical means, for example,heart rate measured by an optical device. Physiological information caninclude, but is not limited to, heart rate, blood pressure, oxygencontent, blood alcohol content (BAC), respiratory rate, perspirationrate, skin conductance, brain wave activity, digestion information,salivation information, among others. Physiological information can alsoinclude information about the autonomic nervous systems of the humanbody derived intrinsically.

Derived intrinsically includes physiological sensors that directlymeasure the internal characteristic of the human body. For example,heart rate sensors, blood pressure sensors, oxygen content sensors,blood alcohol content (BAC) sensors, EEG sensors, FNIRS sensors, FMRIsensors, bio-monitoring sensors, among others. It is understood thatphysiological sensors can be contact sensors and/or contactless sensorsand can include electric current/potential sensors (e.g., proximity,inductive, capacitive, electrostatic), acoustic sensors, subsonic,sonic, and ultrasonic sensors, vibration sensors (e.g., piezoelectric),optical sensors, imaging sensors, thermal sensors, temperature sensors,pressure sensors, photoelectric sensors, among others.

In some embodiments, the ECU 106 can include provisions for receivinginformation about the physiological state of a driver. In oneembodiment, the ECU 106 could receive physiological information relatedto the autonomic nervous system (or visceral nervous system) of adriver. As mentioned above, in one embodiment, the ECU 106 can include aport 178 for receiving physiological information about the state of adriver from a bio-monitoring sensor 180. Examples of differentphysiological information about a driver that could be received from thebio-monitoring sensor 180 include, but are not limited to: heartinformation, such as, heart rate, blood pressure, blood flow, oxygencontent, blood alcohol content (BAC), etc., brain information, such as,electroencephalogram (EEG) measurements, functional near infraredspectroscopy (fNIRS), functional magnetic resonance imaging (fMRI),digestion information, respiration rate information, salivationinformation, perspiration information, pupil dilation information, aswell as other kinds of information related to the autonomic nervoussystem or other biological systems of the driver.

Generally, a bio-monitoring sensor could be disposed in any portion of amotor vehicle. In some cases, a bio-monitoring sensor could be disposedin a location proximate to a driver. For example, in one embodimentshown in FIG. 1A, the bio-monitoring sensor 180 is located within or onthe surface of the vehicle seat 168, more specifically, the seat backsupport 172. In other embodiments, however, the bio-monitoring sensor180 could be located in any other portion of the motor vehicle 100,including, but not limited to: a steering wheel (e.g., the touchsteering wheel 134), a headrest (e.g., the headrest 174), a seat belt(e.g., the seat belt 176), an armrest, dashboard, rear-view mirror aswell as any other location. Moreover, in some cases, the bio-monitoringsensor 180 can be a portable sensor that is worn by a driver, associatedwith a portable device located in proximity to the driver, such as asmart phone (e.g., the portable device 122) or similar device,associated with an article of clothing worn by the driver or integratedinto the body of the driver (e.g. an implant). Further, it isunderstood, that the systems and methods described herein can includeone or more bio-monitoring sensors. Exemplary types and locations ofsensors will be discussed in more detail herein.

In some embodiments, the ECU 106 can include provisions for receivingvarious kinds of optical information about a physiological state of adriver. As mentioned above, in one embodiment, the ECU 106 includes aport 160 for receiving information from one or more optical sensingdevices, such as an optical sensing device 162. The optical sensingdevice 162 could be any kind of optical device including a digitalcamera, video camera, infrared sensor, laser sensor, as well as anyother device capable of detecting optical information. In oneembodiment, the optical sensing device 162 can be a video camera. Inanother embodiment, the optical sensing device 162 can be one or morecameras or optical tracking systems. In addition, in some cases, the ECU106 could include a port 164 for communicating with a thermal sensingdevice 166. The thermal sensing device 166 can be configured to detectthermal information about the physiological state of a driver. In somecases, the optical sensing device 162 and the thermal sensing device 166could be combined into a single sensor.

The optical and thermal sensing devices can be used to monitorphysiological information, for example, heart rate, pulse, blood flow,skin color, pupil dilation, respiratory rate, oxygen content, bloodalcohol content (BAC), among others, from image data. For example, heartrate and cardiac pulse can be extracted and computed in by remote andnon-contact means from digital color video recordings of, for example,the human face as proposed by Poh et al., in “Advancements inNoncontact, Multiparameter Physiological Measurements Using a Webcam,”Biomedical Engineering, IEEE Transactions on, vol. 58, no. 1, pp. 7, 11,January 2011, and “Non-contact, Automated Cardiac Pulse MeasurementsUsing Video Imaging and Blind Source Separation,” Optics Express 18(2010):10762.

Further, image and video magnification can be used to visualize the flowof blood and small motions of the drivers face. This information can beused to extract blood flow rate, pulse rates, and skin color informationas proposed by Wu et al., in “Eulerian Video Magnification for RevealingSubtle Changes in the World,” ACM Trans. Graph. 31, 4, Article 65 (July2012), 8 pages. It is appreciated that other types of physiologicalinformation can be extracted using information from optical and thermalsensing devices, such as oxygen content and blood alcohol content.

Referring now to FIG. 3, an illustration of an embodiment of variousmonitoring systems 300 and sensors that can be associated with the motorvehicle 100 is shown. The monitoring systems 300 ascertain, retrieve,and/or obtain information about a driver, and more particularly, adriver state. In some cases, the monitoring systems are autonomicmonitoring systems. These monitoring systems could include one or morebio-monitoring sensors 180. In one embodiment, the monitoring systems300 and sensors of FIG. 3 can be part of a larger physiologicalmonitoring system and/or a larger behavioral monitoring system(discussed below). Thus, in some embodiments, the monitoring systems 300and sensors of FIG. 3 can monitor and obtain physiological informationand/or behavioral information related to a state of a driver. It isunderstood, that reference to monitoring systems herein, can in someembodiments, refer to the vehicle systems of FIG. 2. For example, thevehicle systems of FIG. 2 can monitor and provide vehicle information.

i. Heart Rate Monitoring Systems, Sensors and Signal Processing

Referring again to FIG. 3, in some embodiments, the motor vehicle 100can include a heart rate monitoring system 302. The heart ratemonitoring system 302 can include any devices or systems for monitoringthe heart information of a driver. In some cases, the heart ratemonitoring system 302 could include heart rate sensors 304, bloodpressure sensors 306, oxygen content sensors 308 and blood alcoholcontent sensors 310, as well as any other kinds of sensors for detectingheart information and/or cardiovascular information. Moreover, sensorsfor detecting heart information could be disposed in any locationswithin the motor vehicle 100 to detect the heart information of thedriver 102. For example, the heart rate monitoring system 302 couldinclude sensors disposed in a dashboard, steering wheel (e.g., thesteering wheel 134), seat (e.g., the vehicle seat 168), seat belt (e.g.,the seat belt 176), armrest or other component to detect the heartinformation of a driver.

In one embodiment, the heart rate sensors 304 of the heart ratemonitoring system 302 includes optical sensing devices 162 and/orthermal sensing devices 166 to sense and provide heart rate information,for example, a heart rate signal indicative of a driver state. Forexample, the optical sensing devices 162 and/or the thermal sensingdevice 166 can provide information (e.g., images, video) of the upperbody, face, extremities, and/or head of a driver or occupant. Heart rateinformation can be extracted from said information, for example, heartinformation can be detected from head movements, eye movements, facialmovements, skin color, skin transparency, chest movement, upper bodymovement, among others. It is understood that the heart rate sensors 304including optical sensing devices 162 and/or thermal sensing devices 166to sense and provide heart rate information can be implemented withother exemplary monitoring systems, sensors and sensor analysisdescribed herein.

A.) Monitoring System for Use with a Vehicle

In one embodiment, the heart rate monitoring system 302 includes heartrate sensors 304 located in specific positions within a vehicle toprovide a signal indicative of a driver state, as discussed in U.S. Pat.No. 8,941,499, filed on Aug. 1, 2011 and issued on Jan. 27, 2015,entitled Monitoring System for use with a Vehicle and Method ofAssembling Same, which is incorporated by reference in its entiretyherein. As will be discussed herein, at least some known heart ratedetections have a low signal-to-noise ratio because the heart ratesignal may be relatively weak and/or because the environmental noise ina vehicle may be relatively high. Accordingly, to accurately determine adriver state, a monitoring system must be configured properly to accountfor these issues. The '499 patent will now be discussed, however, forbrevity, the '499 patent will not be discussed in its entirety.

FIG. 4 illustrates an exemplary monitoring system 400 that includes aseat 402 and a seat belt 404 that is selectively coupleable to seat 402to secure an occupant (not shown) within seat 402. More specifically, inthe exemplary embodiment, seat belt 404 is selectively moveable betweenan engaged configuration (shown generally in FIG. 4), wherein seat belt404 is coupled to seat 402, and a disengaged configuration (not shown),wherein at least a portion of seat belt 404 is uncoupled from seat 402.As described herein, monitoring system 400 is used to monitor a driverof the vehicle. Additionally or alternatively, the monitoring system 400may be configured to monitor any other occupant of the vehicle. It isappreciated that the seat 402 and the components shown in FIG. 4 can beimplemented in the motor vehicle 100 of FIG. 1A. For example, seat 402can be similar to the vehicle seat 168 with similar components discussedherein. The monitoring system 400 can be part of the monitoring systemsshown in FIG. 3, for example a heart rate monitoring system 302.Additionally, the monitoring system 400 can include various sensors forheart rate monitoring, for example, the heart rate sensors 304, theblood pressure sensors 306, the oxygen content sensors 308, and/or theblood alcohol content sensors 310.

In the exemplary embodiment of FIG. 4, the seat 402 includes a lowersupport 406 and a back support 408 that extends generally upward fromlower support 406. The seat 402 can also include a headrest 410 thatextends generally upward from the back support 408. The back support 408includes a seat back surface 412 that is oriented to face a front (notshown) of the vehicle. In the exemplary embodiment, seat belt 404 isselectively extendable across seat back surface 412. More specifically,in the exemplary embodiment, a lap belt portion 414 of seat belt 404 isextendable substantially horizontally with respect to seat back surface412, and a sash belt portion 416 of seat belt 404 is extendablesubstantially diagonally with respect to seat back surface 412.Alternatively, seat belt 404 may be extendable in any direction thatenables the monitoring system 400 to function as described herein

In the exemplary embodiment illustrated in FIG. 4, when the monitoringsystem 400 is used, a first sensor 418 is positioned to detect anoccupant's heart rate and/or blood flow rate. It is understood that thefirst sensor could be the bio-monitoring sensor 180 of FIG. 1A. Morespecifically, in the exemplary embodiment shown in FIG. 4, first sensor418 detects an occupant's heart rate and/or blood flow rate when theoccupant is secured within seat 402 and seat belt 404 is in the engagedconfiguration. For example, in the exemplary embodiment, when seat belt404 is in the engaged configuration, first sensor 418 is positioned inrelative close proximity to the occupant's heart. More specifically, inthe exemplary embodiment, first sensor 418 is coupled to seat belt 404or, more specifically, to seat back surface 412 and/or to sash beltportion 416. Alternatively, first sensor 418 may be positioned in anyother location that enables the monitoring system 400 to function asdescribed herein.

In the exemplary embodiment, first sensor 418 has a passive state, asdescribed above, and an active state. In the exemplary embodiment, firstsensor 418 generates a raw signal (not shown), when in the active state,that is representative of biological data and noise detected and/ormeasured by first sensor 418. More specifically, in the exemplaryembodiment, the raw signal is generated proportional to a mechanicalstress and/or vibration detected by first sensor 418. Moreover, in theexemplary embodiment, first sensor 418 generates an alert signal (notshown), when in the active state, that is detectable by the occupant.For example, in one embodiment, first sensor 418 is used to produce atactile and/or audible signal that may be detected by the occupant. Asused herein, the term “biological data” is used to refer to dataassociated with the occupant's heart rate, blood flow rate, and/orbreathing rate. Biological data can also refer to physiologicalinformation. Moreover, as used herein, the term “noise” is used to referto sensor detections other than biological data.

Furthermore, in the exemplary embodiment, a second sensor 420 ispositioned remotely from first sensor 418. More specifically, in theexemplary embodiment, second sensor 420 is positioned to detect noisethat is substantially similar to noise detected by first sensor 418. Forexample, in the exemplary embodiment, second sensor 420 is coupled toseat belt 404 or, more particularly, to lap belt portion 414 and/or tolower support 406. Alternatively, second sensor 420 may be positioned inany other location that enables the monitoring system 400 to function asdescribed herein.

In the exemplary embodiment, second sensor 420 generates a baselinesignal (not shown) that is representative of noise and, moreparticularly, noise that is substantially similar to noise subjected toand detected by first sensor 418. More specifically, in the exemplaryembodiment, the baseline signal generated is proportional to mechanicalstresses and/or vibrations detected by second sensor 420.

In the exemplary embodiment, first sensor 418 and/or second sensor 420is formed with a thin film (not shown) that is flexible, lightweight,and/or durable. As such, in the exemplary embodiment, the thin film maybe contoured to be generally ergonomic and/or comfortable to theoccupant being monitored by the monitoring system 400. For example, inthe exemplary embodiment, the thin film has a substantially low profilewith a thickness (not shown) that is, for example, less than 600 nm.More particularly, in the exemplary embodiment, the thin film thicknessis between approximately 100 nm and 300 nm. Moreover, in the exemplaryembodiment, the flexibility and durability of the material used enablesfirst sensor 418 and/or second sensor 420 to be embedded in seat 402and/or seat belt 404. Alternatively, the thin film may have anythickness that enables first sensor 418 and/or second sensor 420 tofunction as described herein. In the exemplary embodiment, the thin filmis fabricated from a thermoplastic fluropolymer, such as polyvinylidenefluoride, and poled in an electric field to induce a net dipole momenton first sensor 418 and/or second sensor 420. Alternatively, the thinfilm may be fabricated from any material that enables first sensor 418and/or second sensor 420 to function as described herein.

In some embodiments, the first sensor 418 and/or the second sensor 420can be photoplethysmopgraphy (PPG) sensors that optically sense changesin blood volume and blood composition. Thus, PPG sensors can opticallyobtain a photoplethysmogram of cardiac activity as a volumetricmeasurement of pulsatile blood flow. PPG measurements can be sensed atvarious locations on (e.g., contact sensors) or near (e.g., contactlesssensors) an vehicle occupant's body. In another embodiment shown in FIG.4, the seat 402 can also include one or more sensors and/or sensorarrays. For example, the sensor array 422 can include sensors, indicatedby circular elements, in various configurations and locations within theseat 402. It is understood that the sensor array 422 can include sensorsin other shapes, configurations, and positions than those shown in FIG.4.

In one embodiment, the sensor array 422 includes PPG sensors asdescribed in U.S. application Ser. No. 14/697,593 filed on Apr. 27,2015, which is incorporated by reference herein. Similar to theembodiment described above, the '593 application includes provisions forcapturing and decontaminating PPG signals in a vehicle from the sensorarray 422. For example, the sensor array 422 can sense PPG signals todetermine a driver's physiological state and/or motion artifactsassociated with the driver and/or the vehicle. The PPG signals and themotion artifacts can be processed to provide a true biological signal(i.e., PPG signal). Other embodiments including PPG sensors will bedescribed in more detail herein with reference to FIG. 8.

Referring now to FIG. 5 is a block diagram of an exemplary computingdevice 500 that may be used with monitoring system 400 of FIG. 4. Insome embodiments, the computing device 500 could be integrated with themotor vehicle 100 of FIGS. 1A and 1B, for example, as part of the ECU106. In the exemplary embodiment of FIG. 5, computing device 500determines a state of the occupant based on raw signals generated byfirst sensor 418 and/or baseline signals generated by second sensor 420.More specifically, in the exemplary embodiment, computing device 500receives the raw signal from first sensor 418 and the baseline signalfrom second sensor 420, and generates a desired signal (not shown) afterdetermining a difference between the raw signal and the baseline signal.That is, in the exemplary embodiment, computing device 500 increases asignal-to-noise ratio of the raw signal by canceling and/or removing thebaseline signal, i.e., noise, from the raw signal to generate a desiredsignal that is indicative of substantially only the biological data.

Moreover, in the exemplary embodiment, computing device 500 may beselectively tuned to facilitate increasing the signal-to-noise ratio ofthe raw signal, the baseline signal, and/or the desired signal. Forexample, in the exemplary embodiment, computing device 500 is programmedto impedance match, i.e., tune, the raw signal, the baseline signal,and/or the desired signal based on biological data, environmental data,and/or other data. For example, in the exemplary embodiment, the rawsignal, the baseline signal, and/or the desired signal may be tunedbased on a type of clothing the occupant being monitored is wearing.That is, each clothing type and/or layer can have a respective tunecircuit associated with it that enables a desired signal that isindicative of the biological data to be generated.

In the exemplary embodiment, the computing device 500 determines a stateof the occupant based on the desired signal or, more particularly, thebiological data. More specifically, in the exemplary embodiment,computing device 500 creates a parameter matrix (not shown) thatincludes a plurality of footprints associated with the occupant'sbiological data over time. Generally, the plurality of footprints areindicative of the occupant in an operating state. However, when thebiological data associated with at least one footprint deviates beyond apredetermined threshold from the biological data associated with theother footprints, computing device 500 may determine that the occupantis in a drowsy state. For example, in the exemplary embodiment, a heartrate and/or blood flow rate that is slower and/or is less than anaverage heart rate and/or blood flow rate by a predetermined amount mayindicate drowsiness of the occupant.

In the exemplary embodiment, the computing device 500 includes a memorydevice 502 and a processor 504 that is coupled to memory device 502 forexecuting programmed instructions. The memory device 502 and/or theprocessor 504 can be implemented as the memory 110 and/or the processor108 shown in FIG. 1B. Processor 504 may include one or more processingunits (e.g., in a multi-core configuration). In one embodiment,executable instructions and/or biological data are stored in memorydevice 502. For example, in the exemplary embodiment, memory device 502stores software (e.g., software modules 116 of FIG. 1B) for use inconverting a mechanical stress and/or vibration to a signal. Computingdevice 500 is programmable to perform one or more operations describedherein by programming memory device 502 and/or processor 504. Forexample, processor 504 may be programmed by encoding an operation as oneor more executable instructions and providing the executableinstructions in memory device 502.

Similar to the processor 108 of FIG. 1B, the processor 504 may include,but is not limited to, a general purpose central processing unit (CPU),a graphics processing unit (GPU), a microcontroller, a reducedinstruction set computer (RISC) processor, an application specificintegrated circuit (ASIC), a programmable logic circuit (PLC), and/orany other circuit or processor capable of executing the functionsdescribed herein. The methods described herein may be encoded asexecutable instructions embodied in a computer readable medium,including, without limitation, a storage device, and/or a memory device.Such instructions, when executed by a processor, cause the processor toperform at least a portion of the methods described herein. The aboveexamples are exemplary only, and thus are not intended to limit in anyway the definition and/or meaning of the term processor.

Similar to the memory 110 of FIG. 1B, the memory device 502, asdescribed herein, is one or more devices that enable information such asexecutable instructions and/or other data to be stored and retrieved.Memory device 502 may include one or more computer readable media, suchas, without limitation, dynamic random access memory (DRAM), staticrandom access memory (SRAM), a solid-state disk, and/or a hard disk.Memory device 502 may be configured to store, without limitation,executable instructions, biological data, and/or any other type of datasuitable for use with the systems described herein.

In the exemplary embodiment, the computing device 500 includes apresentation interface 506 that is coupled to processor 504.Presentation interface 506 outputs and/or displays information, such as,but not limited to, biological data and/or any other type of data to auser (not shown). For example, presentation interface 506 may include adisplay adapter (not shown) that is coupled to a display device (notshown), such as a cathode ray tube (CRT), a liquid crystal display(LCD), a light-emitting diode (LED) display, an organic LED (OLED)display, and/or an “electronic ink” display. In some embodiments, thepresentation interface 506 could be implemented on a display of one ofthe visual devices 140 of FIG. 1A.

In the exemplary embodiment, computing device 500 includes an inputinterface 508 that receives input from a user. Input interface 508 canbe similar to user input devices 152 of FIG. 1A. For example, inputinterface 508 receives instructions for controlling an operation of themonitoring system 400 and/or any other type of data suitable for usewith the systems described herein. In the exemplary embodiment, inputinterface 508 is coupled to processor 504 and may include, for example,a keyboard, a pointing device, a mouse, a stylus, a touch sensitivepanel (e.g., a touch pad or a touch screen), a gyroscope, anaccelerometer, a position detector, and/or an audio input interface. Asingle component, such as a touch screen, may function as both a displaydevice of presentation interface 506 and as input interface 508

In the exemplary embodiment, computing device 500 includes acommunication interface 510 coupled to memory device 502 and/orprocessor 504. The communication interface 510 can be similar to thecommunication interface 114 of FIG. 1B. Communication interface 510 iscoupled in communication with a remote device, such as first sensor 418,second sensor 420, and/or another computing device 500. For example,communication interface 510 may include, without limitation, a wirednetwork adapter, a wireless network adapter, and/or a mobiletelecommunications adapter.

In the exemplary embodiment, computing device 500 may be used to enablefirst sensor 418 to generate the alert signal. More specifically, in theexemplary embodiment, computing device 500 may be programmed todetermine whether the alert signal is generated based on at least theraw signal from first sensor 418, the baseline signal from second sensor190, and/or the desired signal generated by computing device 500.Moreover, in the exemplary embodiment, computing device 500 may betransmit a signal to first sensor 418 that enables first sensor 418 totransmit a tactile and/or audible signal that may be detected by theoccupant. The tactile and/or audible signal could be implemented throughthe audio devices 144 and/or the tactile devices 148 of FIG. 1A. Assuch, in the exemplary embodiment, the occupant may be stimulated by thealert signal.

According to the embodiment described above with reference to FIGS. 4and 5, the configuration described herein enables a state of an occupant(e.g., a driver state) to be determined. More specifically, theembodiments described herein facilitate increasing a signal indicativeof an occupant's heart rate or blood flow rate and/or reducing undesirednoise. Moreover, the embodiments described herein are generally moreergonomic and/or more comfortable relative to other known monitoringsystems.

It is appreciated that other exemplary vehicle systems and monitoringsystems, including the sensors, sensor placement, sensor configuration,and sensor analysis, described with reference to FIGS. 4 and 5, can beimplemented with the motor vehicle 100 of FIG. 1, the vehicle systems126 and the monitoring systems of FIG. 3. The exemplary systems andmethods described with reference to FIGS. 4 and 5 can be used to monitorthe driver 102 in the motor vehicle 100 and determine one or more driverstates and/or a combined driver state index, which will be described inmore detail herein.

b.) System and Method for Determining Changes in a Driver State

As discussed above, the heart rate monitoring system 302 can include anydevices or systems for monitoring the heart information of a driver. Inone embodiment, the heart rate monitoring system 302 includes heart ratesensors 304 that facilitate systems and methods for determiningbiological changes in a driver state based on parasympathetic andsympathetic activity levels, as discussed in U.S. Pat. No. 9,420,958,entitled System and Method for Determining Changes in a Body State,which is incorporated by reference in its entirety herein. As will bediscussed, parasympathetic and sympathetic activity levels determinedbased on heart rate information can be used to determine one or moredriver states and subsequently control vehicle systems based in part onthe one or more driver states. The '112 application will now bediscussed, however, for brevity, the '112 application will not bediscussed in its entirety.

Functional or structural variations in cardiac activity (e.g., heartrate information) can indicate biological system activity levels (e.g.,parasympathetic and sympathetic activity levels of the autonomic nervoussystem), which can provide accurate measurements of a driver state or atransition from one driver state to another driver state. FIG. 6illustrates an exemplary computer system 600. In some embodiments, theexemplary computer system 600 can be a heart rate monitoring system 302(FIG. 3). Further, the computer system 600 can be implemented as part ofthe ECU 106 shown in FIG. 1B. Referring again to FIG. 6, the computersystem 600 includes a computing device 602, a processor 604, aninput/output device 606, a memory 608, a communication module 610, and amonitoring system 612. The computer system 600 can include similarcomponents and functionality as the ECU 106 in FIG. 1B and themonitoring systems described in FIG. 3. The monitoring system 612 caninclude and/or communicate with a plurality of sensors 614. Theplurality of sensors 614 can include, for example, heart rate sensors304 (FIG. 3).

Referring again to FIG. 6, the processor 604 includes a signal receivingmodule 616, a feature determination module 618, an intervaldetermination module 620, a derivative calculation module 622 and anidentification module 624, which process data signals and executefunctions as described in further detail herein. The monitoring system612 is configured to monitor and measure monitoring informationassociated with an individual for determining changes in a driver stateof the individual and transmit the information to the computing device602. The monitoring information can include heart rate information. Inother embodiments, the monitoring information can include, but is notlimited to, physical characteristics of the individual (e.g., posture,position, movement) and biological characteristics of the individual(e.g., cardiac activity, such as, heart rate, electrocardiogram (EKG),blood pressure, blood flow, oxygen content, blood alcohol content) andother biological systems of the individual (e.g., circulatory system,respiratory system, nervous system, including the autonomic nervoussystem, or other biological systems). Other types of monitoringinformation can include, environmental information, such as, physicalcharacteristics of the environment in proximity to the individual (e.g.,light, temperature, weather, pressure, sounds). The monitoring system612 can include any system configured to monitor and measure themonitoring information, such as, optical devices, thermal devices,autonomic monitoring devices (e.g., heart rate monitoring devices) aswell as any other kinds of devices, sensors, or systems.

In the illustrated embodiment of FIG. 6, the monitoring system 612includes a plurality of sensors 614 for monitoring and measuring themonitoring information. In some embodiments, the sensors 614 can includeheart rate sensors 304, blood pressure sensors 306, oxygen contentsensors 308, blood alcohol content sensors 310, EEG sensors 320, FNIRSsensors 322, FMRI sensors 324, and other sensors utilized by the vehiclesystems and the monitoring systems of FIGS. 2 and 3. The sensors 614sense a stimulus (e.g., a signal, property, measurement or quantity)using various sensor technologies and generate a data stream or signalrepresenting the stimulus. The computing device 602 is capable ofreceiving the data stream or signal representing the stimulus directlyfrom the sensors 614 or via the monitoring system 612. Althoughparticular sensors are described herein, any type of suitable sensor canbe utilized.

The sensors 614 can be contact sensors and/or contactless sensors andcan include electric current/potential sensors (e.g., proximity,inductive, capacitive, electrostatic), subsonic, sonic, and ultrasonicsensors, vibration sensors (e.g., piezoelectric), optical, photoelectricor oxygen sensors, among others. Generally, the sensors 614 can belocated in any position proximate to the individual or on theindividual, in a monitoring device, such as a heart rate monitor, in aportable device, such as, a mobile device, a laptop or similar devices.The sensors and processing of signals generated by the sensors will bediscussed in more detail with reference to FIG. 7 below. Further, themonitoring system 612 and/or the computing device 602 can receive themonitoring information from the portable device or any other device(e.g., a watch, a piece of jewelry, clothing articles) with computingfunctionality (e.g., including a processor similar to processor 604).The portable device may also contain stored monitoring information orprovide access to stored monitoring information on the Internet, othernetworks, and/or external databases.

As mentioned above, in one embodiment, the monitoring system 612 canmonitor and measure monitoring information associated with a vehicleoccupant (e.g., a driver) in a vehicle, for example, the motor vehicle100 and the driver 102 of FIG. 1A. The monitoring system 612 candetermine changes in a driver state of the occupant and transmit themonitoring information to the ECU 106. The monitoring system 612receives the monitoring information from various sensors. The sensorscan include, for example, the optical sensor 162, the thermal sensor166, and the bio-monitoring sensor 180, which can be included as part ofthe plurality of sensors 614.

As discussed herein, the sensors could be disposed in any portion of themotor vehicle 100, for example, in a location proximate to the driver102. For example, in a location in or on the surface of the vehicle seat168, the headrest 174, the steering wheel 134, among others. In anotherembodiment, the sensors could be located in various positions as shownin FIG. 4 (e.g., the seat 402, the seat belt 404, a lower support 406, aback support 408, a seat back surface 412, a lap belt portion 414, and asash belt portion 416). In other embodiments, however, the sensors couldbe located in any other portion of motor vehicle 100, including, but notlimited to an armrest, a seat, a seat belt, dashboard, rear-view mirroras well as any other location. Moreover, in some cases, the sensor canbe a portable sensor that is worn by the driver 102, associated with aportable device located in proximity to the driver 102, such as a smartphone or similar device (e.g., the portable device 122), or associatedwith an article of clothing worn by the driver 102.

With reference to FIG. 7, a computer-implemented method is shown fordetermining changes in a driver state of an individual. In particular,the method will be described in association with the computer system 600of FIG. 6, though it is to be appreciated that the method could be usedwith other computer systems. Additionally, the method can be modifiedfor alternative embodiments described herein (e.g., the motor vehicle100, FIG. 1A). It is to be appreciated that a driver state herein refersto biological or physiological state of an individual or a transition toanother state. For example, a driver state can be one or more of alert,drowsy, distracted, stressed, intoxicated, other generally impairedstates, other emotional states and/or general health states. (Seediscussion of driver state in Section I). Further, cardiac activity or ameasurement of cardiac activity, as used herein, refers to eventsrelated to the flow of blood, the pressure of blood, the sounds and/orthe tactile palpations that occur from the beginning of one heart beatto the beginning of the next heart beat or the electrical activity ofthe heart (e.g., EKG). Thus, the measurement of cardiac activity canindicate a plurality of cardiac cycles or a plurality of heart beats.

At step 702, the method includes receiving a signal from a monitoringsystem. The signal indicates a measurement of cardiac activity of theindividual over a period of time. In one embodiment, the monitoringsystem 612 is configured to monitor cardiac activity of an individualfrom the plurality of sensors 614. As discussed above, the sensors 614sense a stimulus (e.g., a signal, property, measurement or quantity)using various sensor technologies and generate a data stream or signalrepresenting the stimulus. Specifically, the data stream or signalrepresenting the stimulus is transmitted from the sensors to the signalreceiving module 616, directly or via the monitoring system 612. In theillustrated embodiment, the signal receiving module 616 can be furtherconfigured to process the signal thereby generating a proxy of thesignal in a particular form. It is appreciated that the sensors 614 orthe monitoring system 612 can also perform processing functions.Processing can include amplification, mixing, and filtering of thesignal as well as other signal processing techniques. In one embodiment,upon receiving the signal, the signal is processed into a plurality ofwaveforms, where each one of the waveforms indicates one heartbeat.

Particular sensors will now be described in operation for sensingmonitoring information, specifically, physiological characteristics(e.g., cardiac activity). Although specific sensors and methods ofsensing are discussed herein, it will be appreciated that other sensorsand methods of sensing cardiac activity can be implemented. The sensors614 can be contact sensors and/or contactless sensors and can includeelectric current/potential sensors (e.g., proximity, inductive,capacitive, electrostatic), subsonic, sonic, and ultrasonic sensors,vibration sensors (e.g., piezoelectric), optical, photoelectric oroxygen sensors, among others.

Electric current/potential sensors are configured to measure an amountor change in an electric current, electrical charge or an electricfield. In one embodiment, electric potential sensors can measureelectrical activity of the heart of the individual over a period of time(i.e., an EKG). The electric potential sensors can be contact sensors orcontactless sensors located on or in proximity to the individual.

Sonic sensors are configured to measure sound waves or vibration atfrequencies below human auditory range (subsonic), at frequencies withinhuman auditory range (sonic) or at frequencies above human auditoryrange (ultrasonic). In one embodiment, sonic sensors can measure soundwaves or vibration generated by cardiac activity. In another embodiment,ultrasonic sensors generate high frequency sound waves and evaluate theecho received back by the sensor. Specifically, ultrasonic sensors canmeasure sounds or vibrations produced by the heart. For example, theultrasonic sensors can generate sound waves towards the thoracic region(e.g., in front or back of chest area) of an individual and measure anecho received back by the sensor indicating cardiac activity.

Optical sensors provide image-based feedback and include machine visionsystems, cameras and other optical sensors. Digital signals generated bythe optical sensors include a sequence of images to be analyzed. Forexample, in one embodiment, a camera (e.g., the optical sensor 162, FIG.1A) can generate images of eye movement, facial expressions, positioningor posture of the individual.

Photoelectric sensors use optics and light (e.g., infrared) to detect apresence, a volume or a distance of an object. In one embodiment, thephotoelectric sensors optically obtain a photoplethysmogram (PPG) ofcardiac activity, which is a volumetric measurement of pulsatile bloodflow. As discussed above with FIG. 4, PPG measurements can be sensed atvarious locations on or near an individual's body using, for example,optical and/or light sensors (e.g., near-infrared, infrared, laser). Asdiscussed in U.S. application Ser. No. 14/697,593 filed on Apr. 27, 2015and incorporated here, the optical and/or light sensors can beconfigured to increase or decrease an intensity of light emitted to emita plurality of wavelengths based on the location of the sensors and thetype of measurement that is output by the sensors.

FIG. 8 illustrates a schematic representation of an individual 802 and aPPG analysis computer 804. PPG measurements can be obtained fromdifferent locations of the individual 802, for example, a left ear 806,a right ear 808, a left hand/finger 810, a right hand/finger 812, a leftfoot/toe 814, and a right foot/toe 816. In another embodiment, PPGmeasurements can be obtained from different sensors in the sensor array422 shown in FIG. 4. The measurements can be obtained by photoelectricsensors, optical and/or light sensors near or on the above mentionedlocations and transmitted to the PPG analysis computer 804. The PPGanalysis computer 804 includes provisions for analyzing the PPGmeasurements and comparing PPG measurements obtained from differentlocations of the individual 802. In some embodiments, the monitoringsystem 612 or the processor 604 of FIG. 6 can perform the functions ofthe PPG analysis computer 804. In other embodiments, the methodsdescribed with reference to FIGS. 4 and 5 (e.g., the processor 504)and/or the methods described in the '592 application can perform thefunctions of the PPG analysis computer 804. Further, in otherembodiments, the ECU 106 (e.g., the processor 108) shown in FIG. 1B canperform the functions of the PPG analysis computer 804.

Referring again to FIG. 7, at step 704, the method includes determiningat least one signal feature, wherein the signal feature is a reoccurringevent over the period of time. In one embodiment, the featuredetermination module 618 receives the signal from the signal receivingmodule 616 and determines the signal feature. The signal feature can bea signal or signal waveform (i.e., shape) characteristic. Exemplarysignal features include, but are not limited to, a deflection, a sound,a wave, a duration, an interval, an amplitude, a peak, a pulse, awavelength or a frequency that reoccurs in the signal over the period oftime.

As discussed above, the sensors 614 generate a signal representing thestimulus measured. The signal and the signal features vary depending onthe property (i.e., the physiological, biological, or environmentalcharacteristic) sensed the type of sensor and the sensor technology. Thefollowing are exemplary cardiac waveforms (i.e., signals indicating ameasurement of cardiac activity) with signal features reoccurring over aperiod of time. Although specific waveforms are disclosed with respectto cardiac activity, the methods and systems disclosed herein areapplicable to waveforms and signals associated with other physiologicalor environment characteristics associated with individual foridentifying a driver state or a transition to a driver state.

Referring now to FIG. 9A, a cardiac waveform 902 of an electrical signalrepresenting cardiac activity is illustrated. In particular, the cardiacwaveform 902 represents an EKG waveform 902, which is a graphicalrepresentation of the electrical activity of a heart beat (i.e., onecardiac cycle). As shown in FIG. 9B, it is to be appreciated that an EKGcan include a plot of the variation of the electrical activity over aperiod of time (i.e., multiple cardiac cycles).

Each portion of a heartbeat produces a difference deflection on the EKGwaveform 902. These deflections are recorded as a series of positive andnegative waves, namely, waves P, Q, R, S, and T. The Q, R, and S wavescomprise a QRS complex 904, which indicates rapid depolarization of theright and left heart ventricles. The P wave indicates atrialdepolarization and the T wave indicates atrial repolarization. Each wavecan vary in duration, amplitude and form in different individuals. In anormal EKG, the R wave can be the peak of the QRS complex 904.

Other signal features include wave durations or intervals, namely, PRinterval 906, PR segment 908, ST segment 910 and ST interval 912, asshown in FIG. 9A. The PR interval 906 is measured from the beginning ofthe P wave to the beginning of the QRS complex 904. The PR segment 908connects the P wave and the QRS complex 904. The ST segment 910 connectsthe QRS complex 904 and the T wave. The ST interval 912 is measured fromthe S wave to the T wave. It is to be appreciated that other intervals(e.g., QT interval) can be identified from the EKG waveform 902.Additionally, beat-to-beat intervals (i.e., intervals from one cyclefeature to the next cycle feature), for example, an R-R interval (i.e.,the interval between an R wave and the next R wave), may also beidentified. FIG. 9B illustrates a series of cardiac waveforms over aperiod of time indicated by element 914. In FIG. 9B the R waves areindicated by the peaks 916, 918 and 920. Further, R-R intervals areindicated by elements 922 and 924.

Referring again to FIG. 7, in one embodiment, determining a signalfeature includes determining the signal feature as an R wave of an EKGsignal. For example, the R wave of the EKG waveform 902. It isappreciated that the signal feature could also be one or more waves P,Q, R, S, and T or one or more of the intervals described above.

FIG. 10A illustrates another embodiment of a cardiac waveform 1002 of anacoustic signal representing cardiac activity generated or processedfrom a sensor, for example, a sonic or vibrational sensor. Inparticular, the cardiac waveform 1002 represents the sound of aorticblood flow. The cardiac waveform 1002 can include signal featuressimilar to the cardiac waveform 902. Exemplary signal features caninclude a peak 1004 or another wave duration, peak, feature of thecardiac waveform 1002. Specifically, the signal feature reoccurs in thesignal over a period of time. For example, FIG. 10B illustrates anacoustic signal 1006 having a series of cardiac waveforms (i.e., thecardiac waveform 1002) with a series of peaks 1008, 1010, 1012. Thepeaks 1008, 1010, and 1012 are an exemplary signal feature that reoccursin the acoustic signal 1006 over a period of time. It is appreciatedthat other characteristics of the cardiac waveform 1002 and/or theacoustic signal 1006 can also be identified as a signal feature. Forexample, peak intervals 1014 and 1016.

FIG. 10C illustrates a cardiac waveform 1018 from an optical signalrepresenting a measurement of cardiac activity. The optical signal canbe a photoplethsymograph (PPG) signal generated from a photoelectricsensor, an optical sensor or a PPG device. The cardiac waveform 1018 isa PPG signal representing a measurement of pulsatile blood flow. Thecardiac waveform 1018 can include signal features similar to the cardiacwaveform 902. Exemplary signal features can include a peak 1020 oranother wave duration, peak, feature of the waveform 1018. Specifically,the signal feature reoccurs in the signal over a period of time. Forexample, FIG. 10D illustrates an optical signal 1022 having a series ofcardiac waveforms (i.e., the cardiac waveform 1018) with a series ofpeaks 1024, 1026, 1028. The peaks 1024, 1026, and 1028 are an exemplarysignal feature that reoccurs in the optical signal 1022 over a period oftime. It is appreciated that other characteristics of the cardiacwaveform 1018 and/or the optical signal 1022 can also be identified as asignal feature. For example, peak intervals 1030 and 1032.

Referring back to step 704 of FIG. 7, determining at least one signalfeature may include determining a time occurrence of the signal feature.The time occurrence of each signal feature in the signal may be storedin a memory 608 as a vector. For example, the time occurrence of each Rwave of the EKG signal may be stored and expressed in vector form as:T _(0,i) =t _(0,0) ,t _(0,1 . . .) t _(0,i) where t _(0,i) is the timeof observance of the R wave component of the QRS complex and 0≤i≤N.  (1)

For simplicity, the expressions (1)-(4) discussed herein are withreference to the R wave of the cardiac waveform 902 (EKG waveform) as asignal feature. It is to be appreciated that the signal feature could beany signal feature identified in other types of signals as discussedabove. For example, t_(0,i) could also indicate a time observance of apeak 1004 of a cardiac waveform 1002 or a peak 1020 of a cardiacwaveform 1018. It is also appreciated that each expression may containmultiple elements of calculations derived from a signal. The elementscan be stored, for example in a memory 608, in vector form.

At step 706, the method includes determining a first interval betweentwo successive signal features. In another embodiment, a first intervalis an interval between two successive features of each one of the heartbeats of the signal. Successive features, as used herein, refer tosignal features that follow each other or are produced in succession.For example, a first interval can be an interval between a first R waveand a second R wave of the EKG signal (i.e., R-R interval), where thesecond R wave is the next successive R wave to the first R wave. Withreference to FIG. 9B, a first interval can be an interval 922 measuredfrom the peak 916 and to the peak 918. A first interval can also be aninterval 924 measured from the peak 918 to the peak 920. Thus, it isappreciated that a signal can include a plurality of first intervalsbetween a plurality of signal features.

In another example shown in FIG. 10B, a first interval can be aninterval 1014 measured from the peak 1008 to the peak 1010. A firstinterval can also be an interval 1016 measured from the peak 1010 to thepeak 1012. In another example shown in FIG. 10D, a first interval can bean interval 1030 measured from the peak 1024 to the peak 1026. A firstinterval can also be an interval 1032 measured from the peak 1026 and tothe peak 1028. With respect to the expressions (1)-(2), a plurality offirst intervals for an EKG signal can be expressed in vector form as:T _(1,i) =t _(1,1) ,t _(1,2 . . .) t _(1,i) where t _(1,i) ≡t _(0,1) −t_(0,i−1) and 1≤i≤N.  (2)

At step 708, the method includes determining a second interval betweentwo successive first intervals. In one embodiment, the intervaldetermination module 620 can determine the first interval and the secondinterval. In one example, the second interval is an interval, or adifference, between successive R-R intervals. For example, a secondinterval can be the difference between the absolute value of a first R-Rinterval and the absolute value of a second R-R interval, where thesecond R-R interval is the next successive R-R interval to the first R-Rinterval. With reference to FIG. 9B, the second interval can be adifference between the interval 922 and the interval 924. In anotherexample shown in FIG. 10B, the second interval can be a differencebetween the interval 1014 and the interval 1016. In a further exampleshown in FIG. 10D, the second interval can be a difference between theinterval 1030 and the interval 1032. It is understood that a signal caninclude a plurality of second intervals defined by a plurality of firstintervals. With respect to expressions (1)-(2), this difference can beexpressed in vector form as:T _(2,i) =t _(2,2) ,t _(2,3 . . .) t _(2,i) where t _(2,i) ≡└t _(1,i)┘−└t _(1,i−1)┘ and 2≤i≤N.  (3)

At step 710, the method includes calculating a derivative based on thesecond interval. In one embodiment, the derivative calculation module6022 is configured to calculate the derivative. The derivative can becalculated as the second interval divided by the period of time. Withrespect to expressions (1)-(3), the derivative can be expressed invector form as:

$\begin{matrix}{{T_{3,i} = t_{3,2}},{{t_{3,{3\ldots}}t_{3,i}\mspace{14mu}{where}\mspace{14mu} t_{3,i}} \equiv {\frac{t_{2,i}}{t_{0,i} - t_{0,{i - 2}}}\mspace{14mu}{and}\mspace{14mu} 2} \leq i \leq {N.}}} & (4)\end{matrix}$

At step 712, the method includes identifying changes in the driver statebased on the derivative. The identification module 6024 can beconfigured to manipulate the data from expressions (1)-(4) in variousways to identify patterns and metrics associated with the driver state.In one embodiment, identifying the changes in the driver state furtherincludes extracting a series of contiguous heart rate accelerations ordecelerations based on the derivative. More specifically, the derivativeT₃ of the heart rate can be sorted and flagged according to the sign ofthe derivative T₃. The sign of the derivative indicates whether theheart rate is accelerating or decelerating. Where the sign of thederivative is the same for a given number of successive derivatives(T₃), contiguous periods of heart rate acceleration or deceleration canbe identified. The contiguous periods of heart rate acceleration ordeceleration can correlate to a change in a driver state. In particular,a series of contiguous heart rate accelerations and a series ofcontiguous heart rate decelerations correlate to bursts of sympathetic(S) and parasympathetic (PS) activity respectively. Thus, by sorting andflagging contiguous time periods of heart rate acceleration anddeceleration, driver state changes associated with bursts of S and PSactivity can be identified and sorted.

In another embodiment, identifying changes in the driver state furtherincludes calculating a threshold based on a count of the contiguousheart rate accelerations or decelerations in a particular series. Forexample, a threshold of 7 is associated with 7 contiguous heart rateaccelerations or decelerations.

Accordingly, the above described monitoring system 612 can be anexemplary monitoring system as shown in FIG. 3. In one embodiment,monitoring system 612 can be a heart rate monitoring system 302. Themonitoring system 612 can provide monitoring information, for example, aseries of contiguous heart rate accelerations or decelerations based onthe derivative and/or identification of contiguous periods of heart rateacceleration or deceleration, to determine a driver state. Thesefunctional or structural variations in heart rate information canindicate biological system activity levels (e.g., parasympathetic andsympathetic activity levels of the autonomic nervous system), which canprovide accurate measurements of a driver state or a transition from onedriver state to another driver state

It is appreciated that other exemplary vehicle systems and monitoringsystems, including the sensors, sensor placement, sensor configuration,and sensor analysis, described with reference to FIGS. 6-10, can beimplemented with the motor vehicle 100 of FIG. 1, the vehicle systems126 and the monitoring systems of FIG. 3. The exemplary systems andmethods described with reference to FIGS. 6-10 can be used to monitorthe driver 102 in the motor vehicle 100 and determine one or more driverstates and/or a combined driver state index, which will be described inmore detail herein.

c.) System and Method for Biological Signal Analysis

In one embodiment, the heart rate monitoring system 302 includes heartrate sensors 304 that facilitate systems and methods to acquire a truebiological signal analysis, as discussed in U.S. Pat. No. 9,398,875,entitled A System and Method for Biological Signal Analysis, filed onNov. 7, 2013, which is incorporated by reference in its entirety herein.As will be discussed, indicators of aortic blood flow, average heartrate, heart rate variability, and beat-to-beat interval can be used toinfer levels of sympathetic and parasympathetic nervous system activity.This information can be used to determine one or more driver states. The'710 application will now be discussed, however, for brevity, the '710application will not be discussed in its entirety.

In a vehicle environment, various interfaces exist to determineautonomic tone (e.g., levels of sympathetic and parasympathetic nervoussystem activity) of a driver. For example, an interface can acquiredifferent biological signals (e.g., indicating aortic blood flow,average heart rate, heart rate variability, and beat-to-beat interval.)from a driver and analyze the biological signals to determine anestimation of autonomic tone. The vehicle environment, specifically,noise and vibrations from engine idling, road travel, among othersources, can interfere with the acquisition and analysis of biologicalsignals in the vehicle and therefore influence the estimation ofautonomic tone.

In one embodiment, a system for biological signal analysis includes oneor more multidimensional sensor arrays. Referring now to FIG. 11, asystem 1100 for biological signal analysis can be implemented alone orin combination with a computing device 1102 (e.g., a controller, anavigation system, an infotainment system, etc.). Thus, for example, thecomputing device 1102 can be implemented within the ECU 106 of FIGS. 1Aand 1B, the vehicle systems 126 and/or the monitoring systems of FIG. 3.The computing device 1102 includes a processor 1104, a filter 1106, amemory 1108, a disk 1110 and an input/output (I/O) interface 1112, whichare operably connected for computer communication via a bus 1114 and/orother wired and wireless technologies. It is understood that thesecomponents can be similar to the components of the ECU 106, for example,the processor 108, the memory 110, the disk 112, the communicationinterface 114, and the data bus 118. Accordingly, it is understood thatthe ECU 106 can perform some or all of the functions of the computingdevice 1102.

In one embodiment, the computing device 1102 also includes a multiplexor1116. In one embodiment, the filter 1106 can include the multiplexor1116. In another embodiment, the multiplexor 1116 can be implementedexternally from the filter 1106 and/or the computing device 1102. In afurther embodiment, the I/O interface 1112 can include the multiplexor1116.

In the illustrated embodiment of FIG. 11, the system 1100 also includesa multidimensional sensor array 1118. In another exemplary embodiment,the system 1100 includes more than one multidimensional sensor array.For example, in the illustrated embodiment shown in FIG. 11, thecomputing device 1102 can include a second multidimensional sensor array1120 and a third multidimensional sensor array 1122. It will beappreciated that the systems and methods discussed herein can beimplemented with any number of multidimensional sensor arrays (e.g., twomultidimensional sensor arrays or more than three multidimensionalsensor arrays). Further, although some embodiments and examplesdiscussed herein refer to the multidimensional sensor array 1118, itwill be appreciated that the second multidimensional sensor array 1120and the third multidimensional sensor array 1122 provide similarfunctionality as the multidimensional sensor array 1118. Themultidimensional sensor arrays can include similar functionality and canbe implemented similarly to the sensors and sensing devices included inthe monitoring systems of FIG. 3 and other exemplary monitoring systemsdiscussed herein.

The multidimensional sensor array 1118 will now be described in furtherdetail and with regard to an embodiment associated with a vehicle (e.g.,the motor vehicle 100, FIG. 1A). It should be noted that anotherembodiment could be applied to a seat outside a vehicle, such as a chairor a bed. The multidimensional sensor array 1118 is disposed at aposition for sensing biological data associated with a driver. Forexample, the multidimensional sensor array 1118 could be disposed at aposition on or within the vehicle seat 168 of FIG. 1A. Themultidimensional sensor array 1118 includes a plurality of sensors eachof which are mechanically coupled to a common structural couplingmaterial. FIG. 12 illustrates a top schematic view of an exemplarymultidimensional sensor array generally shown by reference numeral 1200.Similarly, FIG. 13 illustrates an orthographic view of themultidimensional sensor array of FIG. 12

As shown in FIGS. 12 and 13, the multidimensional sensor array 1200includes a plurality of sensors M1, M2, M3 and M4. It will beappreciated that in some embodiments, the multidimensional sensor array1200 can include other numbers of sensors, for example, two sensors ormore than four sensors. In the embodiment illustrated in FIGS. 12 and13, the sensors M1, M2, M3, and M4 are acoustic sensors, for example,microphones. Accordingly, the sensors M1, M2, M3 and M4 are configuredto sense an acoustic measurement (e.g., a stimulus) of biological dataassociated with a person and generate a data stream or a raw data signal(e.g., output) representing the acoustic measurement. Biological datacan include, but is not limited to, data associated with the heart(e.g., aortic blood flow, average heart rate, heart rate variability,and beat-to-beat interval), the lungs (e.g., respiratory rate), andother biological systems of the human body.

In the illustrated embodiments of FIGS. 12 and 13, the sensors M1, M2,M3, and M4 are mechanically coupled to a common structural couplingmaterial 1202. The common structural coupling material 1202 provides aconnection in a non-electrical manner between the sensors M1, M2, M3,and M4. The mechanical coupling allows for distribution of ambientmechanical vibrations (e.g., engine noise, road noise) equally to eachof the sensors M1, M2, M3, and M4. In one embodiment, the commonstructural coupling material 1202 is a circuit board upon which thesensors M1, M2, M3 and M4 are fixed (e.g., via adhesives, bonding,pins). In another embodiment, the common structural coupling material1202 is a bracket or includes one or more brackets upon which thesensors M1, M2, M3 and M4 are fixed (e.g., via adhesives, bonding,pins). It will be appreciated that other materials can be used as thecommon structural coupling material 1202. In particular, other materialswith a high modulus of elasticity and a low density can be used as thecommon structural coupling material 1202.

By mechanically coupling the acoustic sensors M1, M2, M3 and M4 to acommon structural coupling material 1202, ambient mechanical vibrationsfrom, for example, the external environment impacts each sensor M1, M2,M3 and M4 equally. As an illustrative example in the context of avehicle (e.g., FIG. 1A), vibrations from the vehicle environment (e.g.,engine noise, road noise), impact each sensor M1, M2, M3 and M4 equallydue to the mechanical coupling provided by the common structuralcoupling material 1202. When the output (e.g., raw signals) from sensorsM1, M2, M3 and M4 are processed and/or filtered, as will later bediscussed), the vibrations can be eliminated from the raw signals as acommon mode.

As shown in FIGS. 13 and 14, the multidimensional sensor array 1200 hasa geometric center 1204 and a center of mass 1206. The center of mass1206 is located external to an area bounded by the plurality of sensors.Specifically, the sensors M1, M2, M3 and M4, which are mechanicallycoupled to the common structural coupling material 1202, are provided(i.e., positioned) so as to define the center of mass 1206 external tothe area bounded by the plurality of sensors. Specifically, the centerof mass 1206 is located external to an area 1208, which is an areabounded by the sensors M1, M2, M3 and M4. The area 1208 is defined by aposition of each of the plurality of sensors M1, M2, M3 and M4 and ageometric center 1210 of the plurality of sensors M1, M2, M3, and M4. Inone embodiment, the center of mass 1206 is created by a weighted portion1212 of the multidimensional sensor array 1200. The weighted portion1212, in one embodiment, is implemented by a power source (not shown)positioned on the multidimensional sensor array 1200. In a furtherembodiment, the center of mass 1206 is created by providing themultidimensional sensor array in a curved shape configuration (notshown). By providing the center of mass 1206 at a location external tothe geometric center 1210 of the plurality of sensors M1, M2, M3 and M4,the ambient mechanical vibration (i.e., noise) registers in each of theplurality of sensors M1, M2, M3 and M4, in plane (i.e., in phase) withrespect to each other.

More specifically, ambient mechanical vibrations are transferred fromthe vehicle to the multidimensional sensor array 1200. Generally, theambient mechanical vibrations manifest as linear motion along ahorizontal axis (X) direction and a vertical axis (Y) direction of themultidimensional sensor array 1200, and in a rotational motion about thehorizontal axis (X) and the vertical axis (Y) of the multidimensionalsensor array 1200. FIG. 13 illustrates a Y, X and Z axes with respect tothe multidimensional sensor array 1200 and the center of mass 1206. Themechanical coupling with respect to each of the sensors M1, M2, M3, andM4, causes each of the sensors M1, M2, M3, and M4 to move in-phase withregards to the vibrational linear motion.

With regards to the vibrational rotational motion, the positioning ofeach of the sensors M1, M2, M3, and M4 with respect to the center ofmass 1206 will now be discussed in more detail. Rotational motion aboutthe horizontal (X) axis is proportional to the magnitude of thevibration multiplied by the moment arm Y. As shown in FIG. 12, each ofthe sensors M1, M2, M3, and M4 define the geometric center 1210. Themoment arm Y is the vertical distance of the geometric center 1210 fromthe vertical axis (i.e., Y coordinate) of the center of mass 1206.Further, a distance y1 is a vertical distance from an axis of thesensors M3, M4 and the center of mass 1206 and a distance y2 is avertical distance from an axis of the sensors M1, M2 and the center ofmass 1206. By positioning each of the sensors M1, M2, M3 and M4 so thatthe ratio of dy/Y is small, then y1 is approximately equal to y2 and theambient mechanical vibrations registered by each of the sensors M1, M2,M3 and M4 are approximately in phase. The ambient mechanical vibrationscan then be processed using filtering techniques that will be discussedin further detail herein. Additionally, rotational motion about thevertical (Y) axis is proportional to the magnitude of the vibrationmultiplied by a moment arm dx. By positioning each of the sensors M1,M2, M3 and M4 so that dx (i.e. the difference between the geometriccenter 1210 and the axis of each of the sensors) is small, the ambientmechanical vibrations registered by each of the sensors M1, M2, M3 andM4 can also be processed using filtering techniques that will bediscussed in further detail herein.

Accordingly, in the embodiment illustrated in FIGS. 12 and 13, at leastone sensor is positioned along the Y axis with a short and a long momentarm and at least one sensor is positioned along the X axis with an xmoment arm on either side of the Y axis. For example, M1 and M2 arepositioned along the Y axis with a short and a long moment arm and M3and M4 are positioned along the X axis with an x moment arm on eitherside of the Y axis. According to an embodiment described herein, theprocessing of the output of each of the sensors is based on the sensorpairs (i.e., M2, M3 and M1, M4) described above. Specifically, thesensors are positioned so that during processing, which is discussedherein, operational amplification adds the motions with the moment armdx in out of phase combinations. Thus, M1 and M4 are positioned onopposite sides of the Y axis and M2 and M3 are positioned on oppositesides of the Y axis. This allows each additive pair to consist of onesensor moving in each direction about the Y axis with moment arm dxallowing for cancellation using common mode with differentialamplification. If both sensors in a pair are on the same side of the Yaxis, the rotary noise from rotation about the Y axis with moment X willnot cancel with differential amplification but will double insteadbecause they are 180 degrees out of phase before subtraction.

Referring again to FIG. 12, in one embodiment, the multidimensionalsensor array further includes one or more clusters. Each of theplurality of sensors M1, M2, M3, and M4 of the multidimensional sensorarray 1200 can be associated with the one or more clusters. For example,in the illustrated embodiment of FIG. 12, the area 1208 can beconsidered a cluster in which sensors M1, M2, M3, and M4 are associated.In another embodiment, which will be discussed herein, sensors M1 and M3can be associated with a first cluster and sensors M3 and M4 can beassociated with a second cluster. It will be appreciated that themultidimensional sensor array 1200 can include any number of clusters(e.g., one cluster or more than two clusters). The clusters may or maynot be associated with a specific location (e.g., position) of thesensor on the common structural coupling material 1202. Further, theclusters can be predefined and associated with any combination ofsensors.

Non-limiting examples of clusters and sensors associated with saidclusters will now be discussed. In one embodiment, a sensor array,including more than one sensor, can be associated with a cluster. In afurther embodiment, the clusters can be a pattern of sensors or an arrayof sensors (as discussed above). In another embodiment, the clusters arepredefined based on the position of the sensors or the output of thesensors. In an additional embodiment, which will be described herein,the multiplexor 1116, can determine the clusters based on a location ofthe multidimensional sensor array, a location of each sensor in themultidimensional sensor array, and/or the output (e.g., the raw datasignal output) of each sensor. Further, a cluster can be determinedand/or a sensor can be associated with a cluster based on thepositioning of the sensors. In one embodiment, a cluster can include atleast one sensor positioned along the Y axis with a short and longmoment arm and at least one sensor position along the X axis with an xmoment arm on either side of the Y axis. Thus, with reference to FIG.12, a first cluster can include M2, M3 and a second cluster can includeM1, M4. It will be appreciated that other combinations and sensor pairscan be associated with a cluster.

As mentioned above, the multidimensional sensor array 1118 and thesystem 1100 of FIG. 11 can be implemented within a vehicle, for example,the motor vehicle 100 of FIG. 1A. In one embodiment, the system 1100 ofFIG. 11 can be used for biological signal analysis of the driver 102 todetermine an arousal level or autonomic tone of the driver 102. Thearousal level or autonomic tone can be used to determine one or moredriver states. FIG. 14 illustrates a simplified view of the motorvehicle 100, the driver 102, and the vehicle seat 168. Further, FIG. 14illustrates another exemplary embodiment of sensor placement in thevehicle seat 168. For convenience, like numerals in FIGS. 1A and 14represent like elements. As discussed above with FIG. 1A, the driver 102is seated in the vehicle seat 168 of the motor vehicle 100. The vehicleseat 168 includes a lower support 170, a seat back support 172 (e.g., abackrest) and a headrest 174, although other configurations of thevehicle seat 168 are contemplated.

The vehicle seat 168 can also include a seat belt (See, for example, theseat belt 404 of FIG. 4 including a lap belt portion 414 and a sash beltportion 416). In the embodiment illustrated in FIG. 14, the elements1402 a, 1402 b, 1402 c indicate positions for sensing biological dataassociated with the driver 102. Specifically, a multidimensional sensorarray or more than one multidimensional sensor array (e.g., themultidimensional sensor array 1118, the second multidimensional sensorarray 1120 and/or the third multidimensional sensor array 1122 can bedisposed at said positions 1402 a, 1402 b, 1402 c for sensing biologicaldata associated with the driver 102.

In particular, in FIG. 101, the positions 1402 a, 1402 b, 1402 c arelocated within the seat back support 172. However, it will beappreciated, the positions can be in other areas of the vehicle seat 168(e.g., seat belt (not shown)) or around the vehicle seat 168 to allowthe multidimensional sensor array disposed at said position to sensebiological data associated with the driver 102. For example, in oneembodiment, the multidimensional sensor array is disposed at a positionfor sensing biological data associated with a thoracic region of thedriver occupying the vehicle. In FIG. 14, the elements 1404 a, 1404 band 1404 c, indicate thoracic regions of the driver 102. Specifically,the elements 1404 a, 1404 b, and 1404 c indicate an uppercervico-thoracic region, a middle thoracic region and a lowerthoraco-lumbar region respectively of the thorax of the driver 102.Accordingly, in FIG. 14, the element 1402 a indicates a position atwhich a multidimensional sensor array is disposed, wherein the positionis proximate to an upper cervico-thoracic region 1404 a of the driver102. Additionally, the element 1404 b indicates a position at which amultidimensional sensor array is disposed, wherein the position isproximate to a middle thoracic region 1404 b of the driver 102. Further,the element 1402 c indicates a position at which a multidimensionalsensor array is disposed, wherein the position is proximate to a lowerthoraco-lumbar region 1404 c of the driver 102.

It will be appreciated that other positions other than the positions1404 a, 1404 b, and 1404 c can be positions proximate to an uppercervico-thoracic region 1404 a, a middle thoracic region 1404 b, and/ora lower thoraco-lumbar region 1404 c. For example, in one embodiment,the multidimensional sensor array can be located in one or morepositions in a seat belt (not shown) that are proximate to an uppercervico-thoracic region 1404 a, a middle thoracic region 1404 b, and/ora lower thoraco-lumbar region 1404 c of the driver 102. In anotherembodiment, the position can be proximate to an axillary region. Othernumbers of multidimensional sensor arrays disposed in other positions orcombinations of positions can also be implemented.

Further, it will be appreciated that one or more multidimensional sensorarrays can be provided and/or disposed at a position for sensingbiological data based on the biological data and/or the biologicalsignal. Different positions can correlate with specific biological dataor provide the best position for measuring and/or collection of saidbiological data. For example, a multidimensional sensor array disposedat a position proximate to an upper cervico-thoracic region 1404 a canbe utilized to obtain a signal associated with heart rate, while aposition proximate to a lower thoraco-lumbar region 1404 c can beutilized to obtain a signal associated with aortic pulse wave. Thus, forexample, during processing, the multiplexor 1116 (FIG. 11) canselectively retrieve or obtain output from a sensor or amultidimensional sensor array based on the biological data to beobtained, the position of the multidimensional sensor array and/or acluster associated with each sensor.

With regards to processing and analysis, the filter 1106 and themultidimensional sensor array 1118 of FIG. 11, will now be will now bedescribed in detail with reference to FIG. 15, which illustrates anexemplary electric circuit diagram 1500. It will be appreciated thatother electric circuit configurations can be implemented, however, forpurposes of simplicity and illustration, the electric circuit diagram1500 has been organized into a sensing portion 1502 (e.g., amultidimensional sensor array 1118) and a filtering portion 1504 (e.g.,a processor 1104 and/or a filter 1106). Further, the electric circuitdiagram includes a multiplexor 1506 (e.g., the multiplexor 1116 in FIG.11), which can be implemented with the sensing portion 1502 and/or thefiltering portion 1504.

The sensing portion 1502 includes acoustic sensors (i.e., microphones)M1, M2, M3 and M4. Similar to FIG. 12, the sensors M1, M2, M3, and M4are mechanically coupled to a common structural coupling material (notshown in FIG. 15). Although four acoustic sensors are illustrated inFIG. 15, other embodiments can include any number of sensors (e.g., lessthan four or more than four). In the embodiment illustrated in FIG. 15,each acoustic sensor M1, M2, M3 and M4 is biased at one tenth a supplyvoltage by a voltage divider circuit formed from resistors R1 and R2 viapull-up resistors Rp1, Rp2, Rp3, and Rp4. In some embodiments, thevoltage is supplied to the multidimensional sensor array by a standardDC power supply (not shown). As discussed above with FIG. 12, thestandard DC power supply could be implemented as a weighted portion1212. The acoustic sensors M1, M2, M3, and M4 sense an acousticmeasurement indicating biological data associated with a driver. Theacoustic measurement is determined by the voltage drop between thepull-up resistors Rp1, Rp2, Rp3 and R4 and the associated acousticsensor to generate an output (e.g., a raw data signal). For example, Vm1is an output signal indicating a voltage measurement registered by thevoltage drop between M1 and Rp1. Vm2 is an output signal indicating avoltage measurement registered by the voltage drop between M2 and Rp2.Vm3 is an output signal indicating a voltage measurement registered bythe voltage drop between M3 and Rp3. Vm4 is an output signal indicatinga voltage measurement registered by the voltage drop between M4 and Rp4.It will be appreciated that other configurations of voltage biasing andimpedance matching can also be implemented with the methods and systemsdescribed herein. Further, other types of microphones and/or acousticsensors, other than electret condenser microphones, can also beimplemented. For example, other microphones can include but are notlimited to, cardioids, unidirectional, omnidirectional,micro-electromechanical, and piezoelectric. It will be appreciated thatother microphones may require different types of biasing and impedancematching configurations.

In one embodiment, each of the plurality of sensors M1, M2, M3, and M4are associated with one or more clusters. In particular, in FIG. 13, thecluster can include at least one sensor positioned along the Y axis witha short and long moment arm and at least one sensor position along the Xaxis with an x moment arm on either side of the Y axis. Similarly,another cluster can include at least one sensor positioned along the Yaxis with a short and long moment arm and at least one sensor positionalong the X axis with an x moment arm on either side of the Y axis.

In one embodiment, the output signals Vm1, Vm2, Vm3 and Vm4 areprocessed (e.g., via the filtering portion 1504) based on the clustersand/or the positioning of each of the sensors. Specifically, the sensorsM2 and M3 are connected to one half of an operational amplifier Amp1 viaan RC couple R1 and C1. The output signals Vm2 and Vm3 are processed bythe Amp 1. Specifically, in this example, the RC couple provides asingle pole of high pass filtering at a frequency of 0.34 Hz. The Amp1is coupled through an output lead via a parallel RC circuit to produce asecond pole of low pass filtering at 3.4 Hz with a gain of R2/R1=1 V/V.The output of the Amp1 is a summation of the output of M2 and M3, equalto Vm2+Vm3 filtered at 0.34-3.4 Hz.

Similarly, the sensors M1 and M4 are also connected to one half of anoperational amplifier Amp2 via an RC couple R1 and C1. The outputsignals Vm1 and Vm4 are processed by the Amp 2. Specifically, the RCcouple provides a single pole of high pass filtering at a frequency of0.34 Hz. The Amp2 is coupled through an output lead via a parallel RCcircuit to produce a second pole of low pass filtering at 3.4 Hz with again of R2/R1=1 V/V. The output of Amp2 is a summation of the output ofM1, M4, equal to Vm1+Vm4 filtered at 0.34-3.4 Hz.

Further, the output of each operational amplifier Amp1, Amp2 is fed to adifferential bioinstrumentation amplifier Amp3 configured to deliver again of 5000/Rg=50000/10=5000 V/V. The Amp3 can provide noisecancellation of the output of the sensors M1, M2, M3, and M4. Inparticular, and as discussed above with FIG. 99, due to the mechanicalcoupling of the sensors M1, M2, M3 and M4, the positioning of thesensors M1, M2, M3 and M4 and the positioning of the center of mass ofthe multidimensional sensor array, environmental vibrations impact eachsensors M1, M2, M3 and M4 equally. Therefore, the Amp3 can remove theenvironmental vibrations from the output signal of each operationalamplifier Amp1, Amp2, as a common mode. The output signal of thedifferential bioinstrumentation amplifier Amp3 is equal toGX[(Vm2+Vm3)−(Vm1+Vm4)] filtered. The output signal of the differentialbioinstrumentation amplifier Amp3 represents a biological signal thatcan be further analyzed (e.g., by the processor 1104) to determineautonomic tone and a level of impairment of the driver 102. Withreference to FIG. 15, by adding together sensor pairs containing both ashort moment arm y1 and a long moment arm y2 (i.e. Vm2+Vm3 and Vm1+Vm4),the differential effects of the differences in the moment arm becomecommon mode and cancel with differential amplification. Likewise, inchoosing sensor pairs in this fashion, the out of plane motion thatoccurs with rotation about the Y axis with moment arm dx also becomescommon mode and cancels out with differential amplification.

As described above, the filter 1106 can include various amplifiers(Amp1, Amp2, Amp3) for processing. It will be appreciated that othertypes of filters and amplifiers can be implemented with the systems andmethods discussed herein. For example, band pass filters, phasecancelling filters, among others. It addition to amplification, thefilter 1106 can include a multiplexor 1116 for selectively receiving theoutput from each of the plurality of sensors and/or selectivelyforwarding the output from each of the plurality of sensors forprocessing. In one embodiment shown in FIG. 15, multiplexor 1506 canselectively receive and/or obtain an output of a sensor from theplurality of sensors M1, M2, M3, M4 of the multidimensional sensor array1118 for further processing by the Amp1, Amp 2 and/or Amp3 based on apredefined factor. For example, the output can be selected based on aposition of a sensor, a position of the multidimensional sensor array, acluster, a signal to noise ratio of the output, among other factors. Inone embodiment, the multiplexor can selectively receive output from asingle sensor, more than one sensor from a single cluster or more thanone cluster. In another embodiment, the multiplexor 1506 can predefine acluster based on a predefined factor, for example, a position of asensor, a position of a multiplexor, a signal to noise ratio of theoutput, among other factors. In an embodiment including more than onemultidimensional sensor array, the multiplexor 1506 can selectivelyreceive and/or forward output of each of the plurality of sensors fromeach of the multidimensional sensor array for further processing by theAmp1, Amp 2 and or Amp3 based on a predefined factor. For example, aposition of the multidimensional sensor array, a position of a sensor, asignal to noise ratio of the output, among other factors.

Further, in some embodiments, the multiplexor 1506 can selectivelyoutput to, for example, the processor 1104, a biological signal based ona predefined factor for use in algorithms and processes for determiningautonomic tone and/or a level of impairment of the driver 102. Forexample, the biological signal can be outputted based on asignal-to-noise ratio, a biological data type, or a position of themultidimensional sensor array, among others. As can be appreciated,various combinations of output from one or more multidimensional sensorarrays and each of the plurality of sensors are contemplated. Byproviding a multidimensional sensor array with a plurality of sensorsmechanically coupled via a common structural coupling material andprocessing the output of the sensors based on regional differences asdiscussed above with FIG. 15, a high quality biological signal can beobtained in a vehicle while the engine is running. This biologicalsignal can be used to determine one or more driver states as will bediscussed herein.

It is also appreciated that other exemplary vehicle systems andmonitoring systems, including the sensors, sensor placement, sensorconfiguration and sensor analysis, described with reference to FIGS.11-15, can be implemented with the motor vehicle 100 of FIG. 1A, thevehicle systems 126 and the monitoring systems of FIG. 3. The exemplarysystems and methods described with reference to FIGS. 11-15 can be usedto monitor the driver 102 in the motor vehicle 100 and determine one ormore driver states and/or a combined driver state index, which will bedescribed in more detail herein.

ii. Other Monitoring Systems, Sensors and Signal Processing

Referring again to FIG. 3 other exemplary monitoring systems will now bedescribed. The motor vehicle 100 can also include a respiratorymonitoring system 312. The respiratory monitoring system 312 couldinclude any devices or systems for monitoring the respiratory function(e.g. breathing) of a driver. For example, the respiratory monitoringsystem 312 could include sensors disposed in a seat for detecting when adriver inhales and exhales. In some embodiments, the motor vehicle 100could include a perspiration monitoring system 314. The perspirationmonitoring system 314 can include any devices or systems for sensingperspiration or sweat from a driver. In some embodiments, the motorvehicle 100 could include a pupil dilation monitoring system 316 forsensing the amount of pupil dilation, or pupil size, in a driver. Insome cases, the pupil dilation monitoring system 316 could include oneor more optical sensing devices, for example, the optical sensing device162.

Additionally, in some embodiments, the motor vehicle 100 can include abrain monitoring system 318 for monitoring various kinds of braininformation. In some cases, the brain monitoring system 318 couldinclude electroencephalogram (EEG) sensors 320, functional near infraredspectroscopy (fNIRS) sensors 322, functional magnetic resonance imaging(fMRI) sensors 324, as well as other kinds of sensors capable ofdetecting brain information. Such sensors could be located in anyportion of the motor vehicle 100. In some cases, sensors associated withthe brain monitoring system 318 could be disposed in a headrest. Inother cases, sensors could be disposed in the roof of the motor vehicle100. In still other cases, sensors could be disposed in any otherlocations.

In some embodiments, the motor vehicle 100 can include a digestionmonitoring system 326. In other embodiments, the motor vehicle 100 caninclude a salivation monitoring system 328. In some cases, monitoringdigestion and/or salivation could also help in determining if a driveris drowsy. Sensors for monitoring digestion information and/orsalivation information can be disposed in any portion of a vehicle. Insome cases, sensors could be disposed on a portable device (e.g., theportable device 122) used or worn by a driver.

It is understood that the monitoring systems for physiologicalmonitoring can include other vehicle systems and sensors discussedherein, for example, the vehicle systems and sensors discussed inSection II (A) and shown in FIG. 2, the behavioral monitoring systemsdiscussed in Section III (B)(2), the vehicular monitoring systemsdiscussed in Section III (B)(3), and the identification systems andsensors discussed in Section III (B)(4) can be types of monitoringsystems for physiological monitoring. Further, it is appreciated, thatany combination of vehicle systems and sensors, physiological monitoringsystems, behavioral monitoring systems, vehicular monitoring systems,and identification systems can be implemented to determine and/or assessone or more driver states based on physiological information.

2. Behavioral Monitoring Systems and Sensors

Generally, behavioral monitoring systems and sensors include, but arenot limited to, any automatic or manual systems and sensors that monitorand provide behavioral information related to a driver of the motorvehicle 100 (e.g., related to a driver state). The behavioral monitoringsystems can include one or more behavioral sensors for sensing andmeasuring a stimulus (e.g., a signal, a property, a measurement, and/ora quantity) associated with the driver of the motor vehicle 100. In someembodiments, the ECU 106 can communicate and obtain a data streamrepresenting the stimulus from the behavioral monitoring system from,for example, a port. In other words, the ECU 106 can communicate andobtain behavioral information from the behavioral monitoring systems ofthe motor vehicle 100.

Behavioral information includes information about the human body derivedextrinsically. Behavioral information is typically observable externallyto the human eye. For example, behavioral information can include eyemovements, mouth movements, facial movements, facial recognition, headmovements, body movements, hand postures, hand placement, body posture,and gesture recognition, among others.

Derived extrinsically includes sensors that measure externalcharacteristics or movements of the human body. Typically, these typesof sensors are visual and/or camera sensors that observe and measure theexternal characteristic. However, it is understood that behavioralsensors can be contact sensors and/or contactless sensors and caninclude electric current/potential sensors (e.g., proximity, inductive,capacitive, electrostatic), acoustic sensors, subsonic, sonic, andultrasonic sensors, vibration sensors (e.g., piezoelectric), opticalsensors, imaging sensors, thermal sensors, temperature sensors, pressuresensors, photoelectric sensors, among others. It is understood that theabove-mentioned behavioral monitoring systems and sensors can be locatedin various areas of the motor vehicle 100, including, but not limitedto: a steering wheel, dashboard, ceiling, rear-view mirror as well asany other location. Moreover, in some cases the sensors can be aportable sensor that is worn by a driver, associated with a portabledevice located in proximity to the driver, such as a smart phone (e.g.,a camera on a smart phone) or similar device, associated with an articleof clothing worn by the driver or integrated into the body of the driver(e.g. an implant).

In some embodiments, the ECU 106 can include provisions for receivingvarious kinds of optical information about a behavioral state of adriver. In one embodiment, and as discussed above, the ECU 106 caninclude a port 160 for receiving information from one or more opticalsensing devices, such as an optical sensing device 162. The opticalsensing device 162 could be any kind of optical device including adigital camera, video camera, infrared sensor, laser sensor, as well asany other device capable of detecting optical information. In oneembodiment, the optical sensing device 162 can be a video camera. Inanother embodiment, the optical sensing device 162 can be one or morecameras or optical tracking systems, to monitor behavioral information,for example, gestures, head movement, body movement, eye/facialmovement, among others. In addition, in some cases, the ECU 106 couldinclude a port 164 for communicating with a thermal sensing device 166.The thermal sensing device 166 can be configured to detect thermalinformation about a behavioral state of a driver. In some cases, theoptical sensing device 162 and the thermal sensing device 166 could becombined into a single sensor.

Generally, one or more optical sensing devices and/or thermal sensingdevices could be associated with any portion of a motor vehicle. In somecases, an optical sensing device could be mounted to the roof of avehicle cabin. In other cases, an optical sensing device could bemounted in a vehicle dashboard. Moreover, in some cases, multipleoptical sensing devices could be installed inside a motor vehicle toprovide viewpoints of a driver or occupant from multiple differentangles. In one embodiment, the optical sensing device 162 can beinstalled in a portion of the motor vehicle 100 so that the opticalsensing device 162 can capture images of the upper body, face, and/orhead of a driver or occupant. Similarly, the thermal sensing device 166could be located in any portion of the motor vehicle 100 including adashboard, roof or in any other portion. The thermal sensing device 166can also be located to provide a view of the upper body, face and/orhead of a driver.

Referring again to FIG. 3, an illustration of an embodiment of variousmonitoring systems 300 and sensors that could be associated with themotor vehicle 100 is shown. These monitoring systems ascertain,retrieve, and/or obtain information about a driver, and moreparticularly, a driver state. In some cases, the monitoring systems areautonomic monitoring systems. These monitoring systems could include oneor more bio-monitoring sensors 180. In one embodiment, the monitoringsystems and sensors of FIG. 3 can be part of a physiological monitoringsystem and/or a behavioral monitoring system. Thus, in some embodiments,the monitoring systems and sensors of FIG. 3 can monitor and obtainphysiological information and/or behavioral information related to stateof a driver. In one exemplary embodiment, an optical sensing devicecould obtain behavioral information related to the head position oreye/facial movement of the driver. The same optical sensing device couldalso obtain physiological information related to the heart rate of thedriver. Other sensors that obtain both behavioral and physiologicalinformation about the driver are also possible.

In some embodiments, the motor vehicle 100 could include gesturerecognition and monitoring system 330. The gesture recognition andmonitoring system 330 could include any devices, sensors, or systems formonitoring and recognizing gestures of a driver. For example, thegesture recognition and monitoring system 330 could include the opticalsensing device 162, the thermal sensing device 166, and/or othercomputer vision systems to obtain gesture and body information about thedriver and information about the environment of the driver. Thisinformation can be in the form of images, motion measurement, depthmaps, among others. The gesture recognition and monitoring system 330can include gesture recognition and tracking software to recognizegestures, objects, and patterns based on the information. In otherembodiments, the gesture recognition and monitoring system 330 couldalso include provisions for facial recognition and monitoring facialfeatures.

In some embodiments, the motor vehicle 100 could include an eye/facialmovement monitoring system 332. The eye/facial movement monitoringsystem 332 could include any devices, sensors, or systems for monitoringeye/facial movements. Eye movement can include, for example, pupildilation, degree of eye or eyelid closure, eyebrow movement, gazetracking, blinking, and squinting, among others. Eye movement can alsoinclude eye vectoring including the magnitude and direction of eyemovement/eye gaze. Facial movements can include various shape and motionfeatures of the face (e.g., nose, mouth, lips, cheeks, and chin). Forexample, facial movements and parameters that can be sensed, monitoredand/or detected include, but are not limited to, yawning, mouthmovement, mouth shape, mouth open, the degree of opening of the mouth,the duration of opening of the mouth, mouth closed, the degree ofclosing of the mouth, the duration of closing of the mouth, lipmovement, lip shape, the degree of roundness of the lips, the degree towhich a tongue is seen, cheek movement, cheek shape, chin movement, chinshape, etc.

In some embodiments, components of the eye/facial movement monitoringsystem 332 can be combined with components of the gesture recognitionand monitoring system 330 and/or the pupil dilation monitoring system316. The eye/facial movement monitoring system 332 could include theoptical sensing device 162, the thermal sensing device 166, and/or othercomputer vision systems. The eye/facial movement monitoring system 332can also include provisions for pattern recognition and eye/gazetracking.

In some embodiments, the motor vehicle 100 could include a head movementmonitoring system 334. In some embodiments, the ECU 106 can includeprovisions for receiving information about a head pose (i.e., positionand orientation) of the driver's head. The head pose can be used todetermine what direction (e.g., forward-looking, non-forward-looking)the head of the driver is directed to with respect to the vehicle. Insome embodiments described herein, the head pose can be referred to ahead look. In one embodiment, the head movement monitoring system 334provides head vectoring information including the magnitude (e.g., alength of time) and direction of the head pose. In one embodiment, ifthe head pose is forward-looking the driver is determined to be payingattention to the forward field-of-view relative to the vehicle. If thehead pose is non-forward-looking the driver may not be paying attention.Furthermore, the head pose can be analyzed to determine a rotation ofthe head of the driver and a rotation (e.g., head of driver is turned)direction with respect to the driver and the vehicle (i.e., to the left,right, back, forward). It is appreciated that information related to thehead pose and/or head look of the driver received from the head movementmonitoring system 334 can be referred to herein as head movementinformation. Determination of a driver state based on head movementinformation from, for example the head movement monitoring system 334,will be discussed in more detail with reference to FIGS. 16A, 16B, and17.

For reference, FIG. 16A illustrates a side view of a vehicle 1602 with avehicle coordinate system and indication of vehicle pillars A, B, C andD. FIG. 16B is an overhead view of the vehicle 1602 shown in FIG. 16Aincluding a driver 1604 with exemplary head looking directions based onthe head pose with respect to the driver and the vehicle frame. Thevehicle 1602 can be similar to the motor vehicle 100 of FIG. 1A and thedriver 1604 can be similar to the driver 102 of FIG. 1A. Accordingly,the references described with FIGS. 16A, 16B and 17 can be applied tothe motor vehicle 100 and the driver 102 of FIG. 1A.

As shown in FIG. 16B, illustrative head looking directions of the driverare shown with respect to the driver (e.g., the head pose, the bodyposition of the driver, posture) and the vehicle frame (e.g., vehiclecoordinate system, pillars) as: forward-looking, forward rightside-looking, left side-looking, right side-looking, rear leftside-looking, rear right side-looking and rear-looking. It is understoodthat the head looking directions described herein are exemplary innature and could include other head looking directions. Additionally,the head looking directions can be based on different elements of thevehicle frame and/or vehicle and can vary based on the driver's bodypose. Further, in some embodiments, the head looking directions can bemodified based on the driver. For example, identification of the driverand pattern/learning methods of the driver's normative head movements.

In FIG. 16B, the forward-looking direction 1606 is between the left Apillar and the left side of the X-axis of the vehicle. The forward rightside-looking direction 1608 is between the right side of the X-axis ofthe vehicle and the right A pillar. The left side-looking direction 1610is between the left A pillar and a line perpendicular to the driver'sbody (e.g., perpendicular to the head of the driver when the head of thedriver is in a forward-looking direction). The right side-lookingdirection 1612 is between the right A pillar and the line perpendicularto the driver's body (e.g., perpendicular to the head of the driver whenthe head of the driver is in a forward-looking direction). The rear leftside-looking direction 1614 is between the line perpendicular to thedriver's body (e.g., perpendicular to the head of the driver when thehead of the driver is in a forward-looking direction) and the left Bpillar. The rear right side-looking direction 1616 is between the lineperpendicular to the driver's body (e.g., perpendicular to the head ofthe driver when the head of the driver is in a forward-lookingdirection) and the right B pillar. The rear-looking direction 1618 isbetween the right and left B pillars and can include areas around the Cpillars and D pillar.

In some embodiments, the head looking directions shown in FIG. 16B canbe based on a 360 degree axis of rotation between a centroid of thedriver's head and the vehicle frame. Further, it is understood that thehead looking directions can include an angular component, for examplehead tilting up or down (not shown). It is appreciated that thedirections shown in FIG. 16B are exemplary in nature and otherdirections with respect to the vehicle frame can be implemented.Further, it is appreciated that the directions shown in FIG. 16B can bemodified, for example, based on a driver state index and/orcharacteristics and preferences of an identified driver (e.g., driverprofile).

The head pose of the driver, a rotation (e.g., head of driver is turned)direction with respect to the driver and the vehicle (i.e., to the left,right, back, forward) will now be discussed in more detail withreference to FIG. 17. In FIG. 17 a head coordinate frame xyz of thedriver's head is defined as element 1702. Further, a head feature point(e.g., eyes, nose, mouth; not shown) coordinate frame XYZ is defined toa surface having a centroid position at the origin of the coordinatesystem XYZ, where the surface lies within the head coordinate frame xyz.In one embodiment, to determine a rotation and rotation direction of thehead of the driver with respect to the driver, the angular differences(i.e., the rotation and the rotation direction) between the coordinatesystems XYZ and xyz are determined as (αβγ). The angular differences inrelation to a vehicle coordinate system (e.g., the vehicle coordinatesystem shown in FIGS. 16A and 116B) can determine the rotation and therotation direction with respect to the driver and the vehicle. Saiddifferently, the offset orientation between the angular differences andthe vehicle coordinate system describe the rotation and the rotationdirection with respect to the driver and the vehicle. The rotation andthe rotation direction can be realized as head looking directions asshown in FIG. 16B.

Referring again to FIG. 1A, it is understood that in some embodiments,the ECU 106 can include provisions for receiving other types ofinformation about the driver's head. For example, information related tothe distance between a driver's head and a headrest (e.g., via theproximity sensor 184 in the headrest 174). Further, in some embodiments,the motor vehicle 100 can include a body movement monitoring system 336(FIG. 3). For example, the ECU 106 can include provisions for receivinginformation about a body pose (i.e., position and orientation) of thedriver's body in relation to the driver and the vehicle. For example,the information can relate to the posture of the driver's body, arotation of the driver's body, movement of the driver's body, amongothers. In some embodiments, the body movement monitoring system 336provides body and/or body part vectoring information including themagnitude (e.g., a length of time) and direction of the body and/or bodypart.

The information about a head pose and the information about a body posecan be received and determined in various ways, for example, from theoptical sensing device 162 and/or the thermal sensing device 166. Insome embodiments, the head movement monitoring system 334 can includethe optical sensing device 162 and the thermal sensing device 166. Insome embodiments, the body movement monitoring system 336 can includethe optical sensing device 162 and the thermal sensing device 166.

As mentioned above, the optical sensing device 162 could be any kind ofoptical device including a digital camera, video camera, infraredsensor, laser sensor, as well as any other device capable of detectingoptical information. In one embodiment, the optical sensing device 162can be a video camera. In another embodiment, the optical sensing device162 can be one or more cameras or optical tracking systems. The opticalsensing device 162 can sense head movement, body movement, eye movement,facial movement, among others. Moreover, in some cases, multiple opticalsensing devices could be installed inside a motor vehicle to provideviewpoints of a driver or occupant from multiple different angles. Inone embodiment, the optical sensing device 162 can be installed in aportion of the motor vehicle 100 so that the optical sensing device 162can capture images of the upper body, face and/or head of a driver oroccupant. Similarly, the thermal sensing device 166 could be located inany portion of the motor vehicle 100 including a dashboard, roof or inany other portion.

In other cases, information about a position and/or a location of thedriver's head can be received from the proximity sensor 184. Theproximity sensor 184 could be any type of sensor configured to detectthe distance between the driver's head and the headrest 174. In somecases, the proximity sensor 184 could be a capacitor. In other cases,the proximity sensor 184 could be a laser sensing device. In still othercases, any other types of proximity sensors known in the art could beused for the proximity sensor 184. Moreover, in other embodiments, theproximity sensor 184 could be used to detect the distance between anypart of the driver and any portion of the motor vehicle 100 including,but not limited to: a headrest, a seat, a steering wheel, a roof orceiling, a driver side door, a dashboard, a central console as well asany other portion of the motor vehicle 100.

In some embodiments, as discussed above, the motor vehicle 100 caninclude a touch steering wheel system 134. Specifically, the steeringwheel can include sensors (e.g., capacitive sensors, electrodes) mountedin or on the steering wheel. The sensors are configured to measurecontact of the hands, or another appendage of the driver (e.g., arm,wrist, elbow, shoulder, knee) with the steering wheel and a location ofthe contact (e.g., behavioral information). In some embodiments, thesensors are located on the front and back of the steering wheel.Accordingly, the sensors can determine if the driver's hands are incontact with the back of the steering wheel (e.g., gripped and wrappedaround the steering wheel). In one embodiment, the sensors can beconfigured (e.g., positioned) into zones of the steering wheel todetermine where on the steering wheel the appendage is touching. Forexample, the left side of the steering wheel, the right side of thesteering wheel, the left and right side of the steering wheel, the topof the steering wheel, the bottom of the steering wheel, the center ofthe steering wheel, the front of the steering wheel, the back of thesteering wheel, among others.

FIG. 18 illustrates an exemplary touch steering wheel 1802. Capacitivesensors (not shown) can measure the contact and position of the hands1804 and 1806 with respect to the steering wheel 1802. Although handsare shown in contact with the steering wheel 1802 in FIG. 18, it isunderstood that the sensors can measure the contact and position ofother appendages (e.g., wrist, elbow, shoulder, and knee). In thisembodiment, the touch steering wheel 1802 also includes a light bar toprovide visual information to the driver. In some embodiments, thesensors can function as a switch wherein the contact of the hands of thedriver and the location of the contact are associated with actuating adevice and/or a vehicle function of the vehicle. As mentioned above, insome embodiments, the sensors can be configured into zones of thesteering wheel. For example, in FIG. 18, the steering wheel 1802includes a left zone 1810, a right zone 1812, a top zone 1814, a bottomzone 1816, and a center zone 1818. Other zones and configurations ofzones not shown in FIG. 18 can also be implemented. It is understoodthat information about contact and position with respect to the touchsteering wheel 1802 can be referred to herein as hand contactinformation. Other examples of touch steering wheel systems that can beimplemented herein are described in U.S. application Ser. No. 14/744,247filed on Jun. 19, 2015, which is incorporated by reference herein.

It is understood that the monitoring systems for behavioral monitoringcan include other vehicle systems and sensors discussed herein, forexample, the vehicle systems and sensors discussed in Section III (A)and shown in FIG. 2, the physiological monitoring systems discussed inSection III (B) (1), the vehicular monitoring systems discussed inSection III (B) (3), and the identification systems and sensorsdiscussed in Section III (B) (4) can be types of monitoring systems forbehavioral monitoring. Further, it is appreciated, that any combinationof vehicle systems and sensors, physiological monitoring systems,behavioral monitoring systems, vehicular monitoring systems, andidentification systems can be implemented to determine and/or assess oneor more driver states based on behavioral information.

3. Vehicular Monitoring Systems and Sensors

Generally, vehicular monitoring systems and sensors include, but are notlimited to, any automatic or manual systems and sensors that monitor andprovide vehicle information related to the motor vehicle 100 of FIG. 1Aand/or the vehicle systems 126, including those vehicle systems listedin FIG. 2. In some cases, the vehicle information can also be related toa driver of the motor vehicle 100. The vehicular monitoring systems caninclude one or more vehicle sensors for sensing and measuring a stimulus(e.g., a signal, a property, a measurement, or a quantity) associatedwith the motor vehicle 100 and/or a particular vehicle system. In someembodiments, the ECU 106 can communicate and obtain a data streamrepresenting the stimulus from the vehicular monitoring system, thevehicle systems 126 and/or the one or more vehicle sensors via, forexample, the port 128. The data can be vehicle information and/or theECU 106 can process the data into vehicle information and/or process thevehicle information further. Thus, the ECU 106 can communicate andobtain vehicle information from the motor vehicle 100, the vehicularmonitoring systems and/or sensors themselves, the vehicle systems 126and/or sensors themselves, and/or other vehicle sensors, for example,cameras, external radar, and laser sensors, among others.

Vehicle information includes information related to the motor vehicle100 of FIG. 1A and/or the vehicle systems 126, including those vehiclesystems listed in FIG. 2. In some cases, the vehicle information canalso be related to a driver of the motor vehicle 100 (e.g., the driver102). Specifically, vehicle information can include vehicle and/orvehicle system conditions, states, statuses, behaviors, and informationabout the external environment of the vehicle (e.g., other vehicles,pedestrians, objects, road conditions, weather conditions). Exemplaryvehicle information includes, but is not limited to, engine info (forexample, velocity or acceleration), steering information, laneinformation, lane departure information, blind spot monitoringinformation, braking information, collision warning information,navigation information, HVAC information, collision mitigationinformation and automatic cruise control information. Vehicleinformation can be obtained by the ECU 106, the vehicular monitoringsystems themselves, the vehicle systems 126 themselves (e.g., vehiclesystem sensors), or other sensors, for example, cameras, external radarand laser sensors, among others. As will be discussed herein, vehicleinformation can be used by the ECU 106 to determine a vehicular-senseddriver state and/or a vehicular state.

It is understood that the vehicle sensors can include, but are notlimited to, vehicular monitoring system sensors, vehicle system sensorsof the vehicle systems 126 and other vehicle sensors associated with themotor vehicle 100. For example, other vehicle sensors can includecameras mounted to the interior or exterior of the vehicle, radar andlaser sensors mounted to the exterior of the vehicle, external cameras,radar and laser sensors (e.g., on other vehicles in a vehicle-to-vehiclenetwork, street cameras, surveillance cameras). The sensors can be anytype of sensor, for example, acoustic, electric, environmental, optical,imaging, light, pressure, force, thermal, temperature, proximity, amongothers.

Examples of different vehicular monitoring systems, including differentvehicle systems 126 illustrated in FIG. 2 will now be discussed. Itshould be understood that the systems shown in FIG. 2 are only intendedto be exemplary and in some cases, some other additional systems can beincluded. In other cases, some of the systems can be optional and notincluded in all embodiments. Referring again to FIG. 2, the vehicularmonitoring system can include an electronic stability control system 202(also referred to as ESC system 202). The ESC system 202 can includeprovisions for maintaining the stability of the motor vehicle 100. Insome cases, the ESC system 202 can monitor the yaw rate and/or lateral gacceleration of the motor vehicle 100 to help improve traction andstability. The ESC system 202 can actuate one or more brakesautomatically to help improve traction. An example of an electronicstability control system is disclosed in Ellis et al., U.S. Pat. No.8,423,257, filed Mar. 17, 2010, the entirety of which is herebyincorporated by reference. In one embodiment, the electronic stabilitycontrol system can be a vehicle stability system.

In some embodiments, the vehicular monitoring systems can include anantilock brake system 204 (also referred to as an ABS system 204). TheABS system 204 can include various different components such as a speedsensor, a pump for applying pressure to the brake lines, valves forremoving pressure from the brake lines, and a controller. In some cases,a dedicated ABS controller can be used. In other cases, ECU 106 canfunction as an ABS controller. In still other cases, the ABS system 204can provide braking information, for example brake pedal input and/orbrake pedal input pressure/rate, among others. Examples of antilockbraking systems are known in the art. One example is disclosed inIngaki, et al., U.S. Pat. No. 6,908,161, filed Nov. 18, 2003, theentirety of which is hereby incorporated by reference. Using the ABSsystem 204 can help improve traction in the motor vehicle 100 bypreventing the wheels from locking up during braking.

In some embodiments, the vehicular monitoring systems can include abrake assist system 206. The brake assist system 206 can be any systemthat helps to reduce the force required by a driver to depress a brakepedal. In some cases, the brake assist system 206 can be activated forolder drivers or any other drivers who can need assistance with braking.An example of a brake assist system can be found in Wakabayashi et al.,U.S. Pat. No. 6,309,029, filed Nov. 17, 1999, the entirety of which ishereby incorporated by reference.

In some embodiments, the vehicular monitoring systems can include anautomatic brake prefill system 208 (also referred to as an ABP system208). The ABP system 208 includes provisions for prefilling one or morebrake lines with brake fluid prior to a collision. This can helpincrease the reaction time of the braking system as the driver depressesthe brake pedal. Examples of automatic brake prefill systems are knownin the art. One example is disclosed in Bitz, U.S. Pat. No. 7,806,486,filed May 24, 2007, the entirety of which is hereby incorporated byreference.

In some embodiments, the motor vehicle 100 can include an electricparking brake (EPB) system 210. The EPB system 210 includes provisionsfor holding the motor vehicle 100 stationary on grades and flat roads.In particular, the motor vehicle 100 can include an electric park brakeswitch (e.g., a button) that can be activated by the driver 102. Whenactivated, the EPB system 210 controls the braking systems discussedabove to apply to one or more wheels of the motor vehicle 100. Torelease the braking, the driver can engage the electric park brakeswitch and/or press on the accelerator pedal. Additionally, the EPBsystem 210 can include an automatic brake hold control feature thatmaintains brake hold when the vehicle is stopped, even after the brakepedal is released. Thus, when the vehicle comes to a full stop, brakehold is engaged and the brakes continue to hold until the acceleratorpedal is engaged. In some embodiments, the automatic brake hold controlfeature can be manually engaged with a switch. In other embodiments, theautomatic brake hold control feature is engaged automatically.

As mentioned above, the motor vehicle 100 includes provisions forcommunicating and/or controlling various systems and/or functionsassociated with the engine 104. In one embodiment, the engine 104includes an idle stop function that can be controlled by the ECU 106and/or the engine 104 based information from, for example, the engine104 (e.g., automatic transmission), the antilock brake system 204, thebrake assist system 205, the automatic brake prefill system 208, and/orthe EPB system 210. Specifically, the idle stop function includesprovisions to automatically stop and restart the engine 104 to helpmaximize fuel economy depending on environmental and vehicle conditions.For example, the ECU 106 can activate the idle stop feature based ongear information from the engine 104 (e.g., automatic transmission) andbrake pedal position information from the braking systems describedabove. Thus, when the vehicle stops with a gear position in Drive (D)and the brake pedal is pressed, the ECU 106 controls the engine to turnOFF. When the brake pedal is subsequently released, the ECU 106 controlsthe engine to restart (e.g., turn ON) and the vehicle can begin to move.In some embodiments, when the idle stop function is activated, the ECU106 can control the visual devices 140 to provide an idle stop indicatorto the driver. For example, a visual device 140 on a dashboard of themotor vehicle 100 can be controlled to display an idle stop indicator.Activation of the idle stop function can be disabled in certainsituations based on other vehicle conditions (e.g., seat belt isfastened, vehicle is stopped on a steep hill). Further, the idle stopfunction can be manually controlled by the driver 102 using, forexample, an idle stop switch located in the motor vehicle 100.

In some embodiments, the vehicular monitoring systems can include a lowspeed follow system 212 (also referred to as an LSF system 212). The LSFsystem 212 includes provisions for automatically following a precedingvehicle at a set distance or range of distances. This can reduce theneed for the driver to constantly press and depress the accelerationpedal in slow traffic situations. The LSF system 212 can includecomponents for monitoring the relative position of a preceding vehicle(for example, using remote sensing devices such as lidar or radar). Insome cases, the LSF system 212 can include provisions for communicatingwith any preceding vehicles for determining the GPS positions and/orspeeds of the vehicles. Examples of low speed follow systems are knownin the art. One example is disclosed in Arai, U.S. Pat. No. 7,337,056,filed Mar. 23, 2005, the entirety of which is hereby incorporated byreference. Another example is disclosed in Higashimata et al., U.S. Pat.No. 6,292,737, filed May 19, 2000, the entirety of which is herebydisclosed by reference.

In some embodiments, the vehicular monitoring systems can include acruise control system 214. Cruise control systems are well known in theart and allow a user to set a cruising speed that is automaticallymaintained by a vehicle control system. For example, while traveling ona highway, a driver can set the cruising speed to 55 mph. The cruisecontrol system 214 can maintain the vehicle speed at approximately 55mph automatically, until the driver depresses the brake pedal orotherwise deactivates the cruising function.

In some embodiments, the vehicular monitoring systems can include anautomatic cruise control system 216 (also referred to as an ACC system216). In some cases, the ACC system 216 can include provisions forautomatically controlling the vehicle to maintain a predeterminedfollowing distance behind a preceding vehicle or to prevent a vehiclefrom getting closer than a predetermined distance to a precedingvehicle. The ACC system 216 can include components for monitoring therelative position of a preceding vehicle (for example, using remotesensing devices such as lidar or radar). In some cases, the ACC system216 can include provisions for communicating with any preceding vehiclesfor determining the GPS positions and/or speeds of the vehicles. Anexample of an automatic cruise control system is disclosed in Arai etal., U.S. Pat. No. 7,280,903, filed Aug. 31, 2005, the entirety of whichis hereby incorporated by reference.

In some embodiments, the vehicular monitoring systems can include acollision warning system 218. In some cases, the collision warningsystem 218 can include provisions for warning a driver of any potentialcollision threats with one or more vehicles, objects, and/orpedestrians. For example, a collision warning system can warn a driverwhen another vehicle is passing through an intersection as the motorvehicle 100 approaches the same intersection. Examples of collisionwarning systems are disclosed in Mochizuki, U.S. Pat. No. 8,558,718,filed Sep. 20, 2010, and Mochizuki et al., U.S. Pat. No. 8,587,418,filed Jul. 28, 2010, the entirety of both being hereby incorporated byreference. In one embodiment, the collision warning system 218 could bea forward collision warning system, including warning of vehicles and/orpedestrians. In another embodiment, the collision warning system 218could be a cross traffic monitoring system, utilizing backup cameras orback sensors to determine if a pedestrian or another vehicle is behindthe vehicle.

In some embodiments, the vehicular monitoring systems can include acollision mitigation braking system 220 (also referred to as a CMBS220). The CMBS 220 can include provisions for monitoring vehicleoperating conditions (including target vehicles, objects, andpedestrians in the environment of the vehicle) and automaticallyapplying various stages of warning and/or control to mitigatecollisions. For example, in some cases, the CMBS 220 can monitor forwardvehicles using radar or other type of remote sensing device. If themotor vehicle 100 gets too close to a forward vehicle, the CMBS 220could enter a first warning stage. During the first warning stage, avisual and/or audible warning can be provided to warn the driver. If themotor vehicle 100 continues to get closer to the forward vehicle, theCMBS 220 could enter a second warning stage. During the second warningstage, the CMBS 220 could apply automatic seat belt pretensioning. Insome cases, visual and/or audible warnings could continue throughout thesecond warning stage. Moreover, in some cases, during the second stageautomatic braking could also be activated to help reduce the vehiclespeed. In some cases, a third stage of operation for the CMBS 220 caninvolve braking the vehicle and tightening a seat belt automatically insituations where a collision is very likely. An example of such a systemis disclosed in Bond, et al., U.S. Pat. No. 6,607,255, and filed Jan.17, 2002, the entirety of which is hereby incorporated by reference. Theterm collision mitigation braking system as used throughout thisdetailed description and in the claims can refer to any system that iscapable of sensing potential collision threats and providing varioustypes of warning responses as well as automated braking in response topotential collisions.

In some embodiments, the vehicular monitoring systems can include a lanedeparture warning system 222 (also referred to as an LDW system 222).The LDW system 222 can determine when a driver is deviating from a laneand provide a warning signal to alert the driver. Examples of lanedeparture warning systems can be found in Tanida et al., U.S. Pat. No.8,063,754, filed Dec. 17, 2007, the entirety of which is herebyincorporated by reference.

In some embodiments, the vehicular monitoring systems can include ablind spot indicator system 224 (also referred to as a BSI system 224).The blind spot indicator system 224 can include provisions for helpingto monitor the blind spot of a driver. In some cases, the blind spotindicator system 224 can include provisions to warn a driver if avehicle is located within a blind spot. In other cases, the blind spotindicator system 224 can include provisions to warn a driver if apedestrian or other object is located within a blind spot. Any knownsystems for detecting objects traveling around a vehicle can be used.

In some embodiments, the vehicular monitoring systems can include a lanekeep assist system 226 (also referred to as an LKAS system 226). Thelane keep assist system 226 can include provisions for helping a driverto stay in the current lane. In some cases, the lane keep assist system226 can warn a driver if the motor vehicle 100 is unintentionallydrifting into another lane. In addition, in some cases, the lane keepassist system 226 can provide assisting control to maintain a vehicle ina predetermined lane. For example, the lane keep assist system 226 cancontrol the electronic power steering system 132 by applying an amountof counter-steering force to keep the vehicle in the predetermined lane.In another embodiment, the lane keep assist system 226, in, for example,an automatic control mode can automatically control the electronic powersteering system 132 to keep the vehicle in the predetermined lane basedon identifying and monitoring lane markers of the predetermined lane. Anexample of a lane keep assist system is disclosed in Nishikawa et al.,U.S. Pat. No. 6,092,619, filed May 7, 1997, the entirety of which ishereby incorporated by reference.

In some embodiments, the vehicular monitoring systems can include a lanemonitoring system 228. In some embodiments, the lane monitoring system228 could be combined or integrated with the blind spot indicator system224 and/or the lane keep assist system 226. The lane monitoring system228 includes provisions for monitoring and detecting the state of thevehicle, and elements in the environment of the vehicle, for example,pedestrians, objects, other vehicles, cross traffic, among others. Upondetection of said elements, the lane monitoring system 228 can warn adriver and/or work in conjunction with the lane keep assist system 226to assist in maintaining control of the vehicle to avoid potentialcollisions and/or dangerous situations. The lane keep assist system 226and/or the lane monitoring system 228 can include sensors and/or opticaldevices (e.g., cameras) located in various areas of the vehicle (e.g.,front, rear, sides, and roof). These sensors and/or optical devicesprovide a broader view of the roadway and/or environment of the vehicle.In some embodiments, the lane monitoring system 228 can capture imagesof a rear region of a vehicle and a blind spot region of the vehicle outof viewing range of a side mirror adjacent to the rear region of thevehicle, compress said images and display said images to the driver. Anexample of a lane monitoring system is disclosed in Nishiguichi et al.,U.S. Publication Number 2013/0038735, filed on Feb. 16, 2011, theentirety of which is incorporated by reference. It is understood thatafter detecting the state of the vehicle, the lane monitoring system 228can provide warnings or driver assistances with other vehicles systems,for example, the electronic stability control system 202, the brakeassist system 206, the collision warning system 218, the collisionmitigation braking system 220, the blind spot indicator system 224,among others.

In some embodiments, the vehicular monitoring systems can include anavigation system 230. The navigation system 230 could be any systemcapable of receiving, sending and/or processing navigation information.The term “navigation information” refers to any information that can beused to assist in determining a location or providing directions to alocation. Some examples of navigation information include streetaddresses, street names, street or address numbers, apartment or suitenumbers, intersection information, points of interest, parks, anypolitical or geographical subdivision including town, township,province, prefecture, city, state, district, ZIP or postal code, andcountry. Navigation information can also include commercial informationincluding business and restaurant names, commercial districts, shoppingcenters, and parking facilities. In some cases, the navigation systemcould be integrated into the motor vehicle, for example, as a part ofthe infotainment system 154. Navigation information could also includetraffic patterns, characteristics of roads, and other information aboutroads the motor vehicle currently is travelling on or will travel on inaccordance with a current route. In other cases, the navigation systemcould be a portable, stand-alone navigation system, or could be part ofa portable device, for example, the portable device 122.

In some embodiments, the vehicular monitoring systems can include aninfotainment system. As mentioned above, in some embodiments, the visualdevices 140, the audio devices 144, the tactile devices 148 and/or theuser input devices 152 can be part of a larger infotainment system 154.In a further embodiment, the infotainment system 154 can facilitatemobile phone and/or portable device connectivity to the vehicle toallow, for example, the playing of content from the mobile device to theinfotainment system. Accordingly, in one embodiment, the vehicle caninclude a hands free portable device (e.g., telephone) system 232. Thehands free portable device system 232 can include a telephone device,for example integrated with the infotainment system, a microphone (e.g.,audio device) mounted in the vehicle. In one embodiment, the hands freeportable device system 232 can include the portable device 122 (e.g., amobile phone, a smart phone, a tablet with phone capabilities). Thetelephone device is configured to use the portable device, themicrophone, and the vehicle audio system to provide an in-vehicletelephone feature and/or provide content from the portable device in thevehicle. In some embodiments, the telephone device is omitted as theportable device can provide telephone functions. This allows the vehicleoccupant to realize functions of the portable device through theinfotainment system without physical interaction with the portabledevice.

In some embodiments, the vehicular monitoring systems can include aclimate control system 234. The climate control system 234 can be anytype of system used for controlling the temperature or other ambientconditions in the motor vehicle 100. In some cases, the climate controlsystem 234 can comprise a heating, ventilation and air conditioningsystem as well as an electronic controller for operating the HVACsystem. In some embodiments, the climate control system 234 can includea separate dedicated controller. In other embodiments, the ECU 106 canfunction as a controller for the climate control system 234. Any kind ofclimate control system known in the art can be used.

In some embodiments, the vehicular monitoring systems can include anelectronic pretensioning system 236 (also referred to as an EPT system236). The EPT system 236 can be used with a seat belt (not shown) forthe motor vehicle 100. The EPT system 236 can include provisions forautomatically tightening, or tensioning, the seat belt. In some cases,the EPT system 236 can automatically pretension the seat belt prior to acollision. An example of an electronic pretensioning system is disclosedin Masuda et al., U.S. Pat. No. 6,164,700, filed Apr. 20, 1999, theentirety of which is hereby incorporated by reference.

In some embodiments, the vehicular monitoring systems can include avehicle mode selector system 238 that modifies driving performanceaccording to preset parameters related to the mode selected. Modes caninclude, but are not limited to, normal, economy, sport, sport+(plus),auto, and terrain/condition specific modes (e.g., snow, mud, off-road,steep grades). For example, in an economy mode, the ECU 106 can controlthe engine 104 (or vehicle systems related to the engine 104) to providea more consistent engine speed thereby increasing fuel economy. The ECU106 can also control other vehicle systems to ease the load on theengine 104, for example, modifying the climate control system 234. In asport mode, the ECU 106 can control the EPS 132 and/or the ESC system202 to increase steering feel and feedback. In terrain/conditionspecific modes (e.g., snow, mud, sand, off-road, steep grades), the ECU106 can control various vehicle systems to provide handling, and safetyfeatures conducive to the specific terrain and conditions. In an automode, the ECU 106 can control various vehicle systems to provide full(e.g., autonomous) or partial automatic control of the vehicle. It isunderstood that the modes and features of the modes described above areexemplary in nature and that other modes and features can beimplemented. Further it is appreciated that more than one mode could beimplemented at the same or substantially the same time.

In some embodiments, the vehicular monitoring systems can include a turnsignal control system 240 for controlling turn signals (e.g.,directional indicators) and braking signals. For example, the turnsignal control system 240 can control turn signal indicator lamps (e.g.,mounted on the left and right front and rear corners of the vehicle, theside of the vehicle, the exterior side mirrors). The turn signal controlsystem 240 can control (e.g., turn ON/OFF) the turn signal indicatorlamps upon receiving a turn signal input from the driver (e.g., inputvia a user input device 152, a turn signal actuator, etc.). In otherembodiments, the turn signal control system 240 can control a featureand/or a visual cue of the turn signal indicator lamps. For example, abrightness, a color, a light pattern, a mode among others. The featureand/or visual cue control can be based on input received from the driveror can be an automatic control based on input from another vehiclesystem and/or a driver state. For example, the turn signal controlsystem 240 can control the turn signal indicator lamps based on anemergency event (e.g., receiving a signal from the collision warningsystem) to provide warnings to other vehicles and/or provide informationabout occupants in the vehicle. Further, the turn signal control system240 can control braking signals (e.g., braking indicator lamps mountedon the rear of the vehicle) alone or in conjunction with a brakingsystem discussed herein. The turn signal control system 240 can alsocontrol a feature and/or visual cue of the braking signals similar tothe turn signal indicator lamps described above.

In some embodiments, the vehicular monitoring systems can include aheadlight control system 242 for controlling headlamps and/or floodlamps mounted on the vehicle (e.g., located the right and left frontcorners of the vehicle). The headlight control system 242 can control(e.g., turn ON/OFF, adjust) the headlamps upon receiving an input fromthe driver. In other embodiments, the headlight control system 242 cancontrol (e.g., turn ON/OFF, adjust) the headlamps automatically anddynamically based on information from one or more of the vehiclesystems. For example, the headlight control system 242 can actuate theheadlamps and/or adjust features of the headlights based onenvironmental/road conditions (e.g., luminance outside, weather), timeof day, among others. It is understood that the turn signal controlsystem 240 and the headlight control system 242 could be part of alarger vehicle lighting control system.

In some embodiments, the vehicular monitoring systems can include afailure detection system 244 that detects a failure in one or more ofthe vehicle systems 126. More specifically, the failure detection system244 receives information from a vehicle system and executes a fail-safefunction (e.g., system shut down) or a non-fail-safe function (e.g.,system control) based on the information and a level of failure. Inoperation, the failure detection system 244 monitors and/or receivessignals from one or more vehicle systems 126. The signals are analyzedand compared to pre-determined failure and control levels associatedwith the vehicle system. Once the failure detection system 244 detectsthe signals meets a pre-determined level, the failure detection system244 initiates control of the one or more vehicle systems and/or shutsdown the one or more vehicle systems. It is understood that one or moreof the vehicle systems 126 could implement an independent failuredetection system. In some embodiments, the failure detection system 244can be integrated with an on-board diagnostic system of the motorvehicle 100. Further, in some embodiments, the failure detection system244 could determine failure of a vehicle system based on a comparison ofinformation from more than one vehicle system. For example, the failuredetection system 244 can compare information indicating hand and/orappendage contact from the touch steering wheel system 134 and theelectronic power steering system 132 to determine failure of a touchsensor as described in U.S. application Ser. No. 14/733,836 filed onJun. 8, 2015 and incorporated herein by reference.

Additionally, the vehicular monitoring systems can include other vehiclesystems 126 and other kinds of devices, components, or systems used withvehicles. The vehicular monitoring systems can include one of thevehicle systems 126 or more than one of the vehicle systems 126. It willbe understood that each of vehicular monitoring system can be astandalone system or can be integrated with the ECU 106. For example, insome cases, the ECU 106 can operate as a controller for variouscomponents of one or more vehicular monitoring system. In other cases,some systems can comprise separate dedicated controllers thatcommunicate with the ECU 106 through one or more ports.

As mentioned above, in certain embodiments, vehicle systems andmonitoring systems can be used alone or in combination for receivingmonitoring information. For example, in some embodiments, vehicularmonitoring systems, physiological monitoring systems and behavioralmonitoring systems can be used in combination for receiving monitoringinformation. Accordingly, one or more monitoring systems can include oneor more vehicle systems (FIG. 2) and/or one or more monitoring systems(e.g., physiological monitoring systems and/or behavioral monitoringsystems (FIG. 3). For example, in one embodiment, the heart ratemonitoring system 302 including heart rate sensors 304 and vehiclesystems 126 including various vehicle sensors facilitate systems andmethods for determining information transfer rates between a driver anda vehicle, as discussed in U.S. application Ser. No. 14/573,778 filed onDec. 17, 2014, entitled System and Method for Determining TheInformation Transfer Rate Between a Driver and a Vehicle, which isincorporated by reference in its entirety herein. The '020 applicationwill now be discussed, however, for brevity, the '020 application willnot be discussed in its entirety.

To maintain control of a vehicle, a constant flow of information from adriver to a vehicle is required. A reduction in the flow of informationfrom the driver to the vehicle can results in a reduction or loss ofvehicular control. Thus, an accurate determination of flow ofinformation can be used to determine a driver state. FIG. 19 illustratesa schematic view of a vehicle 1900 having an information transfer ratesystem 1902 for determining the information transfer rate between adriver 1904 and vehicle 1900 according to an exemplary embodiment. Thevehicle 1900 can include similar components and functions as the motorvehicle 100 of FIG. 1A. Additionally, the information transfer ratesystem 1902 can be a type of monitoring system and/or obtain informationfrom the vehicle systems 126 and/or the monitoring systems of FIG. 3.

Referring again to FIG. 19, in one embodiment, the vehicle 1900comprises a driver information sensing device 1906, a vehicleinformation sensing device 1908, a driver alert device 1910, a GPS 1912,and optionally an external information sensing device 1914. To controlthe vehicle 1900, the driver 1904 must transmit information by way ofone or more driver control input devices to produce appropriate changesin vehicle acceleration, velocity, lane position, and direction. Drivercontrol input devices (not shown) include, but are not limited to, asteering wheel, accelerator pedal, and brake pedal. Thus, a reduction ininformation transfer from the driver 1904 to the vehicle 1900 can signala reduction in vehicular control, as could be the case with a driver1904 who is distracted, drowsy, intoxicated or experiencing a medicalemergency.

In one embodiment, the driver information sensing device 1906 canmeasure driver information directly from the driver 1904, such asbiometric data and direct driver control input device data. Driverbiometric data can include one or more types of driver biometric data,including, but not limited to, eyelid aperture, pupil diameter, headposition, gaze direction, eye blink rate, respiratory rate, heart rate,hand position, aortic blood flow, leg position, and brain electricalactivity. Direct driver control input device data can include data fromone or more types of driver control input devices, such as, but notlimited to, the steering wheel, brake pedal, and gas pedal of vehicle1900. Accordingly, the direct driver control input device data, caninclude, but is not limited to, one or more of the position of thevehicle steering wheel, turn velocity of the steering wheel, turnacceleration of the steering wheel, position of the vehicle gas pedal,velocity of the gas pedal, acceleration of the gas pedal, position ofthe vehicle brake pedal, velocity of the brake pedal, and accelerationof the brake pedal.

It is contemplated that in some embodiments, one driver informationsensing device 1906 can be used to measure one or more types of driverinformation directly from the driver 1904. In other embodiments,multiple driver information sensing devices 1906 can be used to measuremultiple types of driver information directly from the driver 1904. Forexample, in one embodiment, driver information sensing device 1906 caninclude an electroencephalograph for measuring the driver brainelectrical activity. In another embodiment, one driver informationsensing device 1906 can include a camera for measuring the driver eyelidaperture, the gas pedal for measuring the position of the vehicle gaspedal, and the brake pedal for measuring the position of the vehiclebrake pedal, and so forth.

Further, in other embodiments, the driver information sensing device1906 can be a camera for measuring the driver eyelid aperture, anotherdriver information sensing device 1906 can be a driver control inputdevice, such as the vehicle gas pedal, or a component of the gas pedal,for measuring the position of the gas pedal, and an additional driverinformation sensing device 1906 can be another driver control inputdevice, such as the vehicle brake pedal, or a component of the brakepedal, for measuring the position of the brake pedal. In otherembodiments, driver information sensing device 1906 can be comprised ofone or more of a contact and/or contactless sensors and can includeelectric current/potential sensors (e.g., proximity, inductive,capacitive, electrostatic), subsonic, sonic, and ultrasonic sensors,vibration sensors (e.g., piezoelectric) visual, photoelectric, oxygensensors, as well as any other kinds of devices, sensors, or systems thatare capable of measuring driver information directly from the driver1904.

In one embodiment, the vehicle information sensing device 1908 canmeasure vehicle information directly from a vehicle system of vehicle1900. For example, the vehicle information sensing device 1908 canmeasure vehicle information directly from the vehicle 1900, such as thelane position, lane deviation, linear and angular vehicle position,velocity and acceleration, distance from potential obstacles in frontof, beside and behind the vehicle 1900, reliance on cruise control,reliance on assisted steering and reaction to known obstacles, such asconstruction barricades, traffic signals, and stopped vehicles.

As with the driver information sensing device 1906, in some embodiments,one vehicle information sensing device 1908 can be used to measure oneor more types of vehicle information directly from the vehicle 1900. Inother embodiments, multiple vehicle information sensing devices 1908 canbe used to measure multiple types of vehicle information. For example,in one embodiment, the vehicle information sensing device 1908 caninclude a camera for measuring lane position of the vehicle 1900 and anaccelerometer for measuring the acceleration of the vehicle 1900. Infurther embodiments, the vehicle information sensing device 1908 can bea camera for measuring lane position of the vehicle 1900, anothervehicle information sensing device 1908 of the vehicle 1900 can be anaccelerometer for measuring the acceleration of the vehicle 1900, and athird vehicle information sensing device 1908 can be an ultrasonicdetector for measuring the distance from the vehicle 1900 to anypotential obstacles located around the vehicle 1900.

The driver alert device 1910 is used to alert the driver 1904 if areduction in vehicle control occurs, namely if the driver safety factor,discussed below, does not exceed a predetermined driver safety alertthreshold, discussed below, due to a low information transfer ratebetween the driver 1904 and the vehicle 1900. The driver alert device1910 can be an output device of the vehicle 1900 that outputs a visual,mechanical, or audio signal to alert the driver 1904 to the reduction invehicle control, which would allow the driver 1904 to take action, suchas pulling the vehicle 1900 over, stopping the vehicle 1900, or swervingthe vehicle 1900.

The external information sensing device 1914 can be used to measureinformation external to the vehicle 1900, and thus the flow ofinformation from the driver 1904 to the vehicle 1900 in reaction to theexternal information. The external information sensing device 1914 canmeasure external information, such as, but not limited to, adjacentvehicles, road construction barricades, stopped traffic, animals, andpedestrians. It is contemplated that in some embodiments, one externalinformation sensing device 1914 can be used to measure one or more typesof external information. In other embodiments, multiple externalinformation sensing devices 1914 can be used to measure multiple typesof external information. For example, in one embodiment, the externalinformation sensing device 1914 can include a camera to sense an animalexternal to the vehicle 1900, an inter-vehicular communication systemfor sensing other vehicles adjacent to the vehicle 1900, and anultrasonic proximity sensor for sensing objects near the vehicle 1900.In another embodiment, one external information sensing device 1914 caninclude a camera to sense an animal external to the vehicle 1900,another external information system can include an inter-vehicularcommunication system for sensing other vehicles adjacent to the vehicle1900, and another external information system can include an ultrasonicproximity sensor for sensing objects near the vehicle 1900.

The GPS 1912 can optionally be present in the vehicle 1900 and can beused to obtain the location, weather, and time of day traffic conditionsat the location of the vehicle 1900 for use during the normalizationprocess of the information transfer rate between the driver 1904 and thevehicle 1900, in embodiments of the information transfer rate system1902, which normalize such information. It is recognized thatnormalizing the information transfer rate between the driver 1904 andthe vehicle 1900 can be necessary due to the fact that a higherinformation transfer rate is required to maintain control of vehicle1900 in some driving conditions and a lower information transfer rate isrequired to maintain control of the vehicle 1900 in other drivingconditions. For example, curvy inner city roads during rush hour onsnowy days require a higher information transfer rate from the driver1904 to the vehicle 1900 to maintain control of the vehicle 1900, thanwill long straight desolate roads in fair weather.

Referring now to FIG. 20, there is shown a schematic detailed view of aninformation transfer rate system 1902 for determining the informationtransfer rate between a driver 1904 and vehicle 1900 according to anexemplary embodiment, which will be described with reference to theelements of FIG. 19. The information transfer rate system B535 comprisesa computer processor 2002 and a memory 2004. Note that the informationtransfer rate system 1902 comprises features, such as communicationinterfaces to the driver information sensing device 1906, vehicleinformation sensing device 1908, driver alert device 1910, GPS 1912, andoptional external information sensing device 1914.

The memory 2004 includes an information transfer rate module 2006. Inone embodiment, the information transfer rate module 2006 receivesdriver information measured directly from the driver 1904 from thedriver information sensing device 1906 in the form of a driver timeseries calculated according to the following equation:D _(x) ={d _(x1) ,d _(x2) . . . d _(xN)}  (5)where: D_(x) is a time series, which is an ordered collection of realvalues of driver information measured directly from the driver 1904using the driver information sensing device 1906, and d_(x) is a timeseries segment of a real value of driver information measured directlyfrom the driver 1904 using the driver information sensing device 1906.

Further, the information transfer rate module 2006 receives vehicleinformation measured directly from the vehicle 1900 from the vehicleinformation sensing device 1908 in the form of a vehicle time seriescalculated according to the following equation:V _(y) ={v _(y1) ,v _(y2) . . . v _(yN)}  (6)where: V_(y) is a time series, which is an ordered collection of realvalues of vehicle information measured directly from the vehicle usingthe vehicle information sensing device 1908, and v_(y) is a time seriessegment of a real value of vehicle information measured directly fromthe vehicle using the vehicle information sensing device 1908.

The information transfer rate module 2006 calculates an informationtransfer rate between the driver and vehicle using the vehicleinformation measured directly from the vehicle 1900 by the vehicleinformation sensing device 1908 and the driver information measureddirectly from the driver 1904 by the driver information sensing device1906. The information transfer rate between the driver 1904 and thevehicle 1900 is calculated using conditional and transfer entropies.Conditional entropy quantifies the amount of information needed todescribe the outcome of a random variable Y given that the value ofanother random variable X is known. Further, transfer entropy is anon-parametric statistic measuring the amount of directed(time-asymmetric) transfer of information between two random processes.Transfer entropy from a process X to another process Y is the amount ofuncertainty reduced in future values of Y by knowing the past values ofX given past values of Y. Thus, in one embodiment, the informationtransfer rate system 1902 measures the reduction in uncertainty in V(vehicle) given historical segments of both V and D (driver) withrespect to the reduction of uncertainty in V given only historicalsegments of V. In other words, the information transfer rate system 1902ascertains how much knowing D assists with determining V.

More specifically, in one embodiment, the information transfer ratebetween the driver 1904 and the vehicle 1900 is calculated according tothe following equation:T _(D) _(x) _(→V) _(y) =H(v _(yi) |v _(y(i−t)) ^((l)))−H(v _(yi) |v_(y(i−t)) ^((l)) ,d _(x(i−τ)) ^((k)))  (7)where: T_(D) _(x) _(→V) _(y) is a transfer entropy from a drivermeasurement x to a vehicle measurement y, H(v_(yi)|v_(y(i−t)) ^((l))) isthe conditional entropy between v_(yi) and a prior segment of V_(y) thatis l points long and delayed by t points. Specifically,v _(y(i−t)) ^((l)) ={v _(y(i−t−l+1)) ,v _(y(i−t−l+2)) , . . . ,v_(y(i−t))}, and H(v _(yi) |v _(y(i−t)) ^((l)) ,d _(x(i−τ)) ^((k)))is the conditional entropy between v_(i) and a prior segment of v_(y)further conditioned on a prior segment of D_(x) that is k points longand delayed by τ time points. Specifically,d _(x(i−τ)) ^((k)) ={d _(x(i−τ−k+1)) ,d _(x(i−τ−k+2)) , . . . ,d_(x(i−τ))}.Note that further conditioning of v_(y1) on d_(x(i−τ)) ^((k)) cannotincrease the uncertainty in v_(i) so:H(v _(yi) |v _(y(i−t)) ^((l)))≥H(v _(yi) |v _(y(i−t)) ^((l)) ,d_(x(i−τ)) ^((k))) and T _(D) _(x) _(→V) _(y) is always greater thanzero.

The information transfer rate module 2006 can be configured to use allof the driver information and vehicle information separately or incombination to form various transfer information sums and calculate aninformation transfer rate between the driver and vehicle. For example,in one embodiment, a total information transfer T_(D→V) is calculated bythe information transfer rate module 2006 using the following equation:T _(D→V)=Σ_(x=1) ^(X)Σ_(y=1) ^(Y) H(v _(yi) |v _(y(i−t)) ^((l)))−H(v_(yi) |v _(y(i−t)) ^((l)) ,d _(x(i−τ)) ^((k)))  (8)which is the total sum over every possible combination of all driverinformation measured directly from the driver 1904 (X in total) by thedriver information sensing device 1906 and all vehicle measurementsmeasured directly from the vehicle (Y in total) by the vehicleinformation sensing device 1908 for a total of X*Y individual sums.

In other embodiments, the information transfer rate module 2006 can beconfigured to use only some of the driver information and vehicleinformation separately or in combination to form various transferinformation sums and calculate an information transfer rate between thedriver and vehicle. For example, in one embodiment, a sum of thecombinations of driver information measurements 3 through 5 measureddirectly from the driver 1904 by the driver information sensing device1906 and vehicle measurements 2 through 6 measured directly from thevehicle 1900 by the vehicle information sensing device 1908, representedas T_(D) ³⁻⁵ _(→V) ²⁻⁶ , can be calculated by the information transferrate module 2006 using the following equation:T _(D) ³⁻⁵ _(→V) ²⁻⁶ =Θ_(x=3) ⁵Σ_(y=2) ⁶ H(v _(yi) |v _(y(i−t))^((l)))−H(v _(yi) |v _(y(i−t) ^((l)) ,d _(x(i−τ)) ^((k)))  (9)Thus, as can be seen, the information transfer rate between the driver1904 and the vehicle 1900 is calculated by the information transfer ratemodule 2006 using entropy. More specifically, the transfer rate iscalculated by information transfer rate module 2006, using transferentropy and conditional entropy. Each of equations (5)-(9), discussedabove, provide an information transfer rate between the driver 1904 andthe vehicle 1900 using transfer entropy and conditional entropy.

In some embodiments, information transfer rate module 2006 also uses theexternal measurements, measurements of information external to thevehicle 1900, provided by external information sensing device 1914 tocalculate the information transfer rate between the driver and vehicle.

In some embodiments, the information transfer rate module 2006normalizes the calculated information transfer rate based on at leastone of the type of driver information measured directly from the driver1904 and the driving conditions. The driving conditions include at leastone of a particular road condition, weather condition, time of day, andtraffic condition. Further, in some embodiments, the informationtransfer rate module 2006 also uses information provided by the GPS 1912of the vehicle 1900 to normalize the information transfer rate for thedriving conditions. In one embodiment, the information transfer ratemodule 2006 determines the maximum information transfer rate byadjusting the parameters t, τ, k, l of the above discussed equations(5)-(9) to determine the maximum information transfer rate between thedriver 545 and the vehicle 100. Specifically, in one embodiment, theparameters t, τ, k, l are adjusted based on at least one of a type ofdriver information measured directly from the driver 1904 and thedriving conditions. The driving conditions include at least one of aparticular road condition, a weather condition, a time of day, and atraffic condition.

In some embodiments, the information transfer rates between the driverand vehicle for all driver measurements and all vehicle measurements arecalculated by the information transfer rate module 2006, tracked by theprocessor 2002, and stored in the memory 2004 to establish personalnormatives for each driver 1904 of the vehicle 1900. These personalnormatives are then stored in a baseline information transfer ratedatabase 2008 as baseline information transfer rate values for thedriver 1904, for retrieval and use by a driver safety factor module2010.

In one embodiment, the baseline information transfer rate database 2008contains baseline information transfer rate values for maintainingcontrol of the vehicle 1900. In some embodiments, the baselineinformation transfer rate database 2008 only contains one baselineinformation transfer rate value. In other embodiments, the baselineinformation transfer rate database 2008 contains at least two differentbaseline information transfer rate values for the driver 1904, with eachvalue adjusted for road conditions. Road conditions can include, but arenot limited to, one or more of type of road, weather, time of day, andtraffic conditions.

In one embodiment, the driver safety factor module 2010 calculates adriver safety factor for the driver 1904 of the vehicle 1900 in realtime. The driver safety factor is the ratio of the rate of informationtransfer between the driver and vehicle calculated by the informationtransfer rate module 2006 and the baseline information transfer rateretrieved from the baseline information transfer rate database 2008 bythe driver safety factor module 2010. In the event that baselineinformation transfer rate database 2008 contains multiple baselineinformation transfer rates for the driver 1904 of vehicle 1900, thedriver safety factor module 2010 retrieves the baseline informationtransfer rate that most closely matches the real time road conditionsfor the road on which the vehicle 1900 is travelling.

In one embodiment, a driver alert module 2012 compares the driver safetyfactor calculated by the driver safety factor module 2010 to apredetermined driver safety alert threshold. In the event that thecalculated driver safety factor does not exceed the predetermined driversafety alert threshold, an alert is issued to the driver 1904 using thedriver alert device 1910, as discussed above. The alert signals to thedriver 1904 that the real time information transfer rate between thedriver and vehicle has fallen below the information transfer ratenecessary for the driver 1904 to maintain suitable control of thevehicle 1900 given the present road conditions.

With reference to FIG. 21, a process flow diagram of a method 2100 fordetermining an information transfer rate between a driver 1904 and avehicle 1900 according to an exemplary embodiment is shown. The methodof FIG. 21 will be described with reference to FIGS. 19 and 21, thoughthe method of FIG. 21 can also be used with other systems andembodiments (e.g., the systems of FIGS. 1-3).

In step 2102 of FIG. 21, driver information is measured directly fromthe driver 1904. In one embodiment, this driver information is measuredusing the driver information sensing device 1906, as described above. Instep 2104, vehicle information is measured directly from the vehicle1900. In one embodiment, this vehicle information is measured using thevehicle information sensing device 1908, as described above.

In step 2106, an information transfer rate between the driver 1904 andthe vehicle 1900 is calculated using the driver information measureddirectly from the driver 1904 in step 2102 and the vehicle informationmeasured directly from the vehicle in step 2104. In one embodiment, thisinformation transfer rate is calculated using the information transferrate module 2006, as described above. Thus, as can be seen, theinformation transfer rate between the driver 1904 and the vehicle 1900is calculated using entropy. More specifically, in some embodiments, thetransfer rate is calculated, using transfer entropy and conditionalentropy, as is shown above in each of equations (7) to (9).

At step 2108, a baseline information transfer rate is retrieved from thebaseline information transfer rate database 2008 by the driver safetyfactor module 2010. As was stated above, in one embodiment, the baselineinformation transfer rate database 2008 contains baseline informationtransfer rate values for maintaining vehicular control. In someembodiments, the baseline information transfer rate database 2008 onlycontains one baseline information transfer rate value. In otherembodiments, the baseline information transfer rate database 2008contains at least two different baseline information transfer ratevalues for the driver 1904, with each value adjusted for roadconditions. Road conditions can include, but are not limited to, one ormore of type of road, weather, time of day, and traffic conditions. Inthe event that the baseline information transfer rate database 2008 hasmultiple information transfer rates for the driver 1904 of vehicle 1900,the driver safety factor module 2010 retrieves the baseline informationtransfer rate that most closely matches the real time road conditionsfor the road on which the vehicle 1900 is travelling.

In step 2110, once the baseline information transfer rate is retrievedfrom the baseline information transfer rate database 2008, the driveralert module 2012 is armed. Information transfer rate system 1902 armsdriver alert module 2012 after a baseline information transfer rate isretrieved from the baseline information transfer rate database 2008 bythe driver safety factor module. Upon arming, driver alert module 2012is prepared to compare a predetermined driver safety alert threshold,stored in memory 2004, to the driver safety factor calculated by thedriver safety factor module 2010. Driver alert module 2012 performs thecomparison when the driver safety factor calculated by the driver safetyfactor module 2010 is provided to the driver alert module 2012 by driversafety factor module 2010.

At step 2112, a driver safety factor is calculated. In one embodiment,the driver safety factor is the ratio of the calculated rate ofinformation transfer to a predetermined information transfer rate. Inone embodiment, the driver safety factor is calculated by the driversafety factor module 2010, as described above, using the informationtransfer rate calculated in step 2106 and the baseline informationtransfer rate retrieved from the baseline information transfer ratedatabase 2008 in step 2108.

In step 2114, the driver safety factor calculated in step 2112 iscompared to a predetermined driver safety alert threshold. In oneembodiment, this comparison is performed by the driver alert module2012, as described above. In step 2116, the driver 1904 is alerted ifthe driver safety factor value does not exceed the predetermined driversafety alert threshold value. The driver safety factor and predetermineddriver safety alert threshold data type can be, but is not limited to,numeric, non-numeric, discrete, or continuous. In one embodiment, if thecomparison made by the driver alert module 2012 in step 2114 indicatesthat the driver safety factor does not exceed the predetermined driversafety alert threshold, then the driver 1904 is alerted using the driveralert device 1910, as described above. Accordingly, an accuratemeasurement of information transfer from the driver to the vehicle canbe monitored and this measurement can be used to determine a driverstate (e.g., a safety factor) to provide accurate warnings to the driverand/or modify control of vehicles according to the driver state.

As discussed in conjunction with FIGS. 1A, 1B, 2, and the motor vehicle100, the vehicle systems 126 and the exemplary monitoring systems caninclude various sensors and sensing devices. Exemplary sensors andsensing devices will now be discussed in more detail. These exemplarysensors and sensing devices are applicable to the vehicle systems ofFIG. 2 and the monitoring systems of FIG. 3, as well as the othermonitoring systems discussed herein. As discussed in more detail above,the sensors can be contact sensors and/or contactless sensors and caninclude electric current/potential sensors (e.g., proximity, inductive,capacitive, electrostatic), subsonic, sonic, and ultrasonic sensors,vibration sensors (e.g., piezoelectric) visual, photoelectric or oxygensensors, among others. The sensors can be configured to sense,physiological, biometric, behaviors parameters of the driver and orparameters related to the vehicle and vehicle systems.

Additionally, the sensors and/or sensing devices can be organized indifferent configurations and/or disposed in one or more positions. Forexample, the sensors could be integrated into a seat, door, dashboard,steering wheel, center console, roof, or any other portion of the motorvehicle 100. In other cases, however, the sensors could be portablesensors worn by a driver, integrated into a portable device carried bythe driver, integrated into an article of clothing worn by the driver(e.g., a watch, a piece of jewelry, clothing articles) or integratedinto the body of the driver (e.g. an implant). Additionally, the sensorscan be located in any position proximate to the individual or on theindividual, in a monitoring device, such as a heart rate monitor, in aportable device, such as, a mobile device, a laptop or similar devices.Further, the monitoring device (e.g., a portable device) may alsocontain stored monitoring information or provide access to storedmonitoring information on the Internet, other networks, and/or externaldatabases.

As discussed above, the sensors could be disposed in any portion of themotor vehicle 100, for example, in a location proximate to the driver102. For example, a proximity sensor 184 is located in the headrest 174.In another embodiment, the bio-monitoring sensor 180 is located in thevehicle seat 168. In a further embodiment, a sensor (not shown) could belocated on or in the steering wheel 134. In other embodiments, however,the sensors could be located in any other portion of motor vehicle 100,including, but not limited to an armrest, dashboard, seat, seat belt,rear-view mirror, as well as any other location.

Further, the sensors, the sensing devices and/or the vehicle systems andmonitoring systems, can process and analyze the stimulus sensed from thesensor and/or the sensing device in various ways to generate a datastream or signal representing the sensed stimulus. In some embodiments,the stimulus sensed is processed according to the location of thesensors and/or the sensing device. In other embodiments, the stimulussensed is processed based on the quality of the data or processed basedon what type of stimulus is being sensed. Other configurations ofprocessing and analysis can also be implemented.

It is understood that monitoring systems for vehicular monitoring caninclude other vehicle systems and sensors discussed herein, for example,the vehicle systems and sensors discussed in Section III (A) and shownin FIG. 2, the physiological monitoring systems discussed in Section III(B)(1), the behavioral monitoring systems discussed in Section III(B)(2), and the identification systems and sensors discussed in SectionIII (B)(4) can be types of monitoring systems for physiologicalmonitoring. Further, it is appreciated, that any combination of vehiclesystems and sensors, physiological monitoring systems, behavioralmonitoring systems, vehicular monitoring systems, and identificationsystems can be implemented to determine and/or assess one or more driverstates based on vehicle information.

4. Identification Systems and Sensors

In some embodiments, the systems and sensors discussed above as well asthe methods and systems for responding to driver state discussed hereincan identify a particular driver to monitor information about thedriver. Further, identification of the driver can provide customized ornormative baseline data for a particular driver. Thus, in oneembodiment, the monitoring systems of FIG. 3 can be used for personalidentification of the driver. In particular, the heart rate monitoringsystem 302 can include any devices or systems for monitoring the heartinformation of a driver. In one embodiment, the heart rate monitoringsystem 302 includes heart rate sensors 304 that facilitate systems andmethods for personal identification of a driver, as discussed in U.S.Pat. No. 9,272,689, entitled System and Method for BiometricIdentification in a Vehicle, which is incorporated by reference in itsentirety herein. The '899 application will now be discussed, however,for brevity, the '899 application will not be discussed in its entirety.

Referring now to FIG. 22, a computer system 2200 for personalidentification of an individual, specifically, of a vehicle occupant(e.g., a driver, one or more passengers) is shown. As will be describedin further detail below, the biometric identification systems andmethods described herein can be utilized in conjunction with saidvehicle systems to provide entry, access, activation, control andpersonalization or modification of said vehicle systems and associateddata.

The computer system 2200 includes a computing device 2202communicatively coupled to a monitoring system 2204 and a plurality ofvehicle systems 2206. It is appreciated that the ECU 106 of FIGS. 1A and1B can include similar components and executed functions similar to thecomputing device 2202. For example, the ECU 106 includes a plurality ofvehicle systems 126 and monitoring systems 300 (FIG. 3). Further, thecomputer system 2200 can be implemented within a vehicle for example,the motor vehicle 100 of FIG. 1A, and can include and/or communicatewith similar components and systems of the motor vehicle 100 (e.g., thevehicle system 126).

The monitoring system 2204 can include and/or communicate with varioussensors. Specifically, with reference to FIG. 1A, the sensors caninclude a first sensor (e.g., a proximity sensor 184) in a headrest 174,a second sensor (e.g., a bio-monitoring sensor 180) in a vehicle seat168. A touch steering wheel 134 may also include sensors (not shown) foridentifying driver state changes. Further, the monitoring system 2204can include and/or communicate with optical and image sensors, forexample, a camera (e.g., an optical sensor 162).

The vehicle systems 2206 can also include data storage mechanism (e.g.,memory) for storing data utilized by said vehicle systems, for example,sensitive data such as contact data, route data, password data, vehicleoccupant profiles, driver behavior profiles, email, among others. Aswill be described in further detail below, the biometric identificationsystems and methods described herein can be utilized in conjunction withsaid vehicle systems to provide entry, access, activation, control andpersonalization or modification of said vehicle systems and associateddata.

Referring again to FIG. 22, the monitoring system 2204 is configured tomonitor and measure monitoring information associated with an individualand transmit the information to the computing device 2202. Themonitoring information can be used to determine biometric identificationof a vehicle occupant and thereby control the vehicle (i.e., entry,access, activation, personalization, and modification of vehiclesystems) based on biometric identification. It is appreciated that themonitoring information and the biometric identification disclosed hereincan be utilized with other systems associated with the vehicle and thevehicle occupant, including, but not limited to, vehicle systems 126,wellness and distraction systems or modifications of such systems basedon the biometric identification.

In the illustrated embodiment, the monitoring system 2204 includes aplurality of sensors 2208 for monitoring and measuring the monitoringinformation. The sensors 2208, sense a stimulus (e.g., a signal,property, measurement or quantity) using various sensor technologies andgenerate a data stream or signal representing the stimulus. Thecomputing device 2202 is capable of receiving the data stream or signalrepresenting the stimulus directly from the sensors 2208 or via themonitoring system 2204. As discussed above, various types of sensors,sensor configurations, sensor placement and analysis can be utilized. Inone embodiment, the monitoring system 2204 and/or the sensors 2208 caninclude a transceiver (not shown) for transmitting a signal towards avehicle occupant and receiving a reflected signal after transmitting thesignal from the vehicle occupant. The transceiver can include one ormore antennas (not shown) to facilitate transmission of the signal andreception of the reflected signal.

With reference to FIG. 23, a computer implemented method is shown foridentifying a vehicle occupant (e.g., a driver 102 of FIG. 1A). Indifferent embodiments, the various steps of the method can beaccomplished by one or more different systems, devices or components. Insome cases, the steps may be accomplished by the ECU 106 of FIG. 1Bincluding the processor 108. For each method discussed and illustratedin the figures, it will be understood that in some embodiments one ormore of the steps could be optional. For purposes of reference, themethod of FIG. 23 will be discussed with components shown in FIGS. 1A,1B, 2, 3, and 22. Moreover, cardiac activity or a measurement of cardiacactivity, as used herein, refers to events related to the flow of blood,the pressure of blood, the sounds and/or the tactile palpations thatoccur from the beginning of one heart beat to the beginning of the nextheart beat or the electrical activity of the heart (e.g., EKG).

At step 2302, the method includes receiving a signal from a plurality ofsensors. The signal can indicate a measurement of cardiac activity, forexample, the signal can be a cardiac signal representing one or more ofa heart beat or a heart rate of the vehicle occupant. In one embodiment,discussed in detail below, the method includes transmitting a signaltowards the vehicle occupant and receiving a reflected signal, thereflected signal indicating a measurement of cardiac activity. It isappreciated that the monitoring system 2204 can be configured to monitorcardiac activity of a vehicle occupant from the plurality of sensors1088 and facilitate transmission of signals to the computing device2202.

The plurality of sensors 2208 are operative to sense a biologicalcharacteristic (e.g., cardiac activity) of the vehicle occupant in thevehicle utilizing contact sensors, contactless sensors, or both contactand contactless sensors. As discussed above, in one embodiment, a sensorcan receive a signal indicating a measurement of cardiac activityproduced by the vehicle occupant upon direct contact of the sensor tothe vehicle occupant. In another embodiment, a sensor can sense a fieldchange (e.g., magnetic, radio frequency) and/or receive a signal (e.g.,signal reflection) indicating a measurement of cardiac activity producedby the vehicle occupant without direct contact of the sensor to thevehicle occupant. I

In particular, the method for identifying a vehicle occupant can furtherinclude a sensor that produces a field or transmits a signal towards thevehicle occupant. The sensors can sense a change in the field producedby the vehicle occupant or receive a reflected signal produced by thevehicle occupant after the signal reflects from the vehicle occupant.Specifically, a sensor can be configured to transmit a signal towards athoracic region (i.e., general chest and/or back area near the heart) ofthe vehicle occupant. The reflected signal can indicate cardiacactivity, for example, a cardiac signal. Signal reflection and magneticand/or electric field sensing sensor technology can be utilized withdifferent types of signals and sensors, as discussed above, and include,but are not limited to, electric current/potential sensors and/or sonicsensors, among others.

In the illustrated embodiment, the receiving module 2218 can be furtherconfigured to process the signal thereby generating a proxy of thesignal in a particular form. It is appreciated that the sensors 2208 orthe monitoring system 2204 can also perform processing functions.Processing can include amplification, mixing, and filtering of thesignal as well as other signal processing techniques known in the art.Processing can also include modifying or converting the signal into aform allowing identification of biometric features. For example, thesignal can be processed into a cardiac waveform, an electrocardiograph(EKG) waveform, or a proxy of an EKG waveform for identificationanalysis.

As discussed above, the sensors 2208 generate a signal representing thestimulus measured. The signal and the signal features vary depending onthe property (i.e., the physiological, biological, or environmentalcharacteristic) sensed the type of sensor and the sensor technology.FIGS. 9A, 9B, 10A, 10B, 10C, 10D, discussed above, are exemplary cardiacwaveforms with signal features reoccurring over a period of time.

Referring specifically to FIGS. 9A and 9B, it is shown that each portionof a heartbeat produces a difference deflection on the EKG waveformA400. These deflections are recorded as a series of positive andnegative waves, namely, waves P, Q, R, S, and T. The Q, R, and S wavescomprise a QRS complex 904, which indicates rapid depolarization of theright and left heart ventricles. The P wave indicates atrialdepolarization and the T wave indicates atrial repolarization. Each wavecan vary in duration, amplitude and form in different individuals. InFIG. 9B the R waves are indicated by the peaks 916, 918 and 920. Thesewaves and wave characteristics, or a combination thereof, can beidentified as signal features for biometric identification.

Other signal features include wave durations or intervals, namely, PRinterval 906, PR segment 908, ST segment 910 and ST interval 912, asshown in FIG. 9A. The PR interval 906 is measured from the beginning ofthe P wave to the beginning of the QRS complex 904. The PR segment 908connects the P wave and the QRS complex 904. The ST segment 910 connectsthe QRS complex 904 and the T wave. The ST interval 912 is measured fromthe S wave to the T wave. It is to be appreciated that other intervals(e.g., QT interval) can be identified from the EKG waveform 902.Additionally, beat-to-beat intervals (i.e., intervals from one cyclefeature to the next cycle feature), for example, an R-R interval (i.e.,the interval between an R wave and the next R wave), may also beidentified. FIG. 9B illustrates a series of cardiac waveforms over aperiod of time indicated by element 914. In FIG. 9B the R waves areindicated by the peaks 916, 918 and 920. Further, R-R intervals areindicated by elements 922 and 924.

Referring back to FIG. 23 and step 2304, the method further includesdetermining a biomarker based on biometric features of the signal. Thebiometric features can include characteristics (i.e. signal features)analyzed, identified, and/or extracted from the signal. The biomarkermodule 2220 can be configured to determine the biomarker. For example,biometric features of a cardiac waveform (e.g., the cardiac waveformsillustrated in FIGS. 9A, 9B, 10A, 10B, 10C, 10 can include waves P, Q,R, S and T or a series of said waves. Other characteristics can includeintervals, time duration of characteristics, and wave amplitude amongothers. The biomarker uniquely identifies the vehicle occupant and canbe any combination of biometric features extracted from the signal. Thebiomarker may include comparisons of one or more of wave amplitude,form, and duration as well as ratios of these features for one wavecompared to another wave. The biomarker is a unique identificationfeature of a vehicle occupant and thereby provides ultra-security andauthorization when used in conjunction with vehicle systems describedherein. It is appreciated that other information can be used alone or incombination with the biometric features of the signal to determine abiomarker. For example, other information can include, but is notlimited to, the psychological and environmental information received andor monitored by the monitoring system 1084. For example, facial featureextraction data (acquired by and optical sensor 162).

Further, in the case where multiple cardiac waveforms are obtained for avehicle occupant, analysis of the heartbeat over time (i.e.,beat-to-beat analysis, heart rate variability) can be performed and usedto obtain the biometric features and/or a biomarker. For example, heartrate variability analysis methods known in the art include time-domainmethods, geometric methods, frequency-domain methods, non-linearmethods, and long term correlations. Different metrics can be derivedusing these methods. For example, a beat-to-beat standard deviation(SDNN), a square root of the mean squared difference of successivebeat-to-beat intervals (RMSSD), a set of R-R intervals, among others.

At step 2306, the method includes identifying the vehicle occupant. Forexample, the identification module 2222 can compare the biomarkeridentified at step 2304 to a stored biomarker in the memory 2214associated with the vehicle occupant. The biomarker may also be storedand accessed via the portable device 122 (FIG. 1A). In anotherembodiment, the identification module 2222 can identify the vehicleoccupant by comparing the biometric features with stored biometricfeatures stored in a personal identification profile associated with thevehicle occupant in the memory 2214 or accessed via the communicationmodule 2216 (e.g., an external database via a network). The storedbiometric features or the biomarker can be based on the signal andacquired prior to using the system for personal identification. Forexample, the biomarker module 2220 can collect baseline metrics from thevehicle occupant during a vehicle learning mode. A biomarker orbiometric features that uniquely identify the vehicle occupant, asdiscussed above, can be determined and stored in the memory 2214 forfuture use with the above described methods and systems. For example,the biomarker module 22220 can then save the biomarker in a personalidentification profile associated with the vehicle occupant.

At step 2308, the identification can be transmitted by the communicationmodule 2216 to one of the plurality of vehicle systems 2206 and access,entry, activation, control, and personalization or modification of thevehicle systems 2206 can be implemented based on the identification. Inanother embodiment, the communication module 2216 can transmit theidentification to an external database or to a portable device. In oneexemplary use of biometric identification, entry to a vehicle (e.g.,vehicle door lock/unlock) is granted to a driver based on the biometricidentification. For example, the computer system 2200, and in particularthe computing device 2202 and the monitoring system 2204 and/or thesensors 2208 can be integrated with a portable device (e.g., theportable device 122) or a key fob. The sensors 2208 can detect a changein an electric field produced by the vehicle occupant indicating ameasurement of cardiac activity (e.g., an EKG) via the key fob outsideof the vehicle. In another embodiment, the sensors 2208 in the key fobcould transmit and receive a reflected signal from a driver in proximityto the portable device or the key fob outside of the vehicle. Thecomputing device 2202 can determine a biomarker based on the signal andidentify the driver based on the biomarker as described above inrelation to the method of FIG. 23. Once the identity of the driver isknown, entry to the vehicle can be granted or denied (e.g., vehicle doorlock/unlock).

Once an identification of the driver and/or vehicle occupant isdetermined, the identification can be utilized in conjunction with othervehicle systems for activation of the vehicle systems or personalizationand modification of the vehicle systems. In one example, collisionmitigation, braking systems, driver assistance systems and algorithmsused therein, can be modified based on the identification to provide atailored driving experience to the driver and/or the vehicle occupant.Further, pattern learning machine algorithms can be used to track dataassociated with an identified driver and the pattern learning can beused to modify different vehicle systems and parameters as discussedherein. In some embodiments, the driver can be associated with a user(e.g., driver) profile including parameters, data, and data trackedovertime specific to the driver. This user profile can be used by thevehicle systems for operation based on the identified user. In oneembodiment, the ECU 106 can store the user profile at the memory 110and/or disk 112 shown in FIG. 1A.

In another embodiment, identification of a driver can be used todetermine a driver state as will be discussed herein. For example,information stored in the identified driver's user profile can becompared to monitoring information to determine a driver state. As anillustrative example, stored steering information in the user profilecan be compared to steering information received from the touch steeringwheel system 134. This comparison can provide an indication of driverstate.

Other known driver identification methods can also be used to identify adriver and thus enable the customization and personalization of one ormore vehicle systems. For example, methods such as facial recognition,iris recognition, and fingerprint recognition could be used. Further,the data used for driver identification can be stored and/or receivedfrom external devices such as a portable device 122 (e.g., a smartphone,a smart watch). Further, it is appreciated that other vehicle systemsand data associated with said vehicle systems can be controlled and/oroperated based on the identification. Moreover, the identification couldbe transmitted to an application (i.e., a telematics application, aportable device application). Biometric identification, as discussedherein, provides a unique, accurate, and secure measurement for entry,access, control, activation, personalization and modification of variousvehicle systems and vehicle system data. In addition, by identifying thedriver, the physiological information, behavioral information andvehicle information can be collected for that particular driver tomodify control parameters, control coefficients and thresholds as willbe discussed in more detail in Section IV (B) (2).

It is appreciated that the systems, sensors, and sensor analysisdiscussed above can be used alone and/or in combination to obtain andassess information about a vehicle and a driver state. The systems andmethods described below for determining one or more driver states canutilized one or more of the above mentioned systems, sensors and sensoranalysis to obtain information to determine the one or more driverstates, including vehicle information, physiological information andbehavioral information, among others.

It is understood that identification systems and sensors can includeother vehicle systems and sensors discussed herein, for example, thevehicle systems and sensors discussed in Section III (A) and shown inFIG. 2, the physiological monitoring systems discussed in Section III(B) (1), the behavioral monitoring systems discussed in Section III (B)(2), and the vehicular monitoring systems discussed in Section III (B)(3) can be types of identification systems. Further, it is appreciated,that any combination of vehicle systems and sensors, physiologicalmonitoring systems, behavioral monitoring systems, vehicular monitoringsystems, and identification systems can be implemented to determineand/or assess one or more driver states based on identificationinformation.

IV. Determine One or More Driver States

A motor vehicle can include provisions for assessing the state of adriver and automatically adjusting the operation of one or more vehiclesystems in response to the driver state or a level of the driver state.As discussed above in detail in Section I above, a “driver state,” canrefer to a measurement of a state of the biological being and/or a stateof the environment of the biological being (e.g., a vehicle). A driverstate or alternatively a “being state” can be one or more of alert,vigilant, drowsy, inattentive, distracted, stressed, intoxicated, othergenerally impaired states, other emotional states and/or general healthstates, among others. Throughout this specification, drowsiness and/ordistractedness will be used as the example driver state being assessed.However, it is understood that any driver state could be determined andassessed, including but not limited to, drowsiness, attentiveness,distractedness, vigilance, impairedness, intoxication, stress, emotionalstates and/or general health states, among others.

In some embodiments, the motor vehicle can include provisions forassessing one or more states of a driver and automatically adjusting theoperation of one or more vehicle systems in response to the one or moredriver states or one or more levels of the driver states. Specifically,the systems and methods for responding to driver state discussed hereincan include determining and/or assessing one or more driver states basedon information from the systems and sensors discussed in Section IIand/or III above.

In one embodiment, a response system can receive information about thestate of a driver and automatically adjust the operation of one or morevehicle systems. As mentioned above with reference to FIG. 1A, forpurposes of convenience, various components, alone or in combination,discussed above, can be referred to herein as the response system 188.In some cases, the response system 188 comprises the ECU 106 as well asone or more sensors, components, devices or systems discussed above. Insome cases, the response system 188 can receive input from variousdevices related to the state of a driver. In some cases, thisinformation is monitoring information as discussed above in Section III(B). The response system 188 can use this information to modify theoperation of one or more of the vehicle systems 126. Moreover, it willbe understood that in different embodiments, the response system 188could be used to control any other components or systems utilized foroperating the motor vehicle 100.

As mentioned briefly above, the response system 188 can includeprovisions for determining one or more driver states. The driver statecan be based on physiological information, behavioral information and/orvehicle information. For example, the response system 188 could detect adriver state for a driver by analyzing heart information, breathing rateinformation, brain information, perspiration information, as well as anyother kinds of autonomic information. Additionally, the response system188 could detect a driver state for a driver by analyzing informationfrom one or more vehicle systems and/or one or more monitoring systems.Further, in some embodiments, the response system 188 could determineone or more driver states and a combined driver state based on the oneor more driver states.

The following detailed description discusses a variety of differentmethods for operating vehicle systems in response to a driver state. Indifferent embodiments, the various different steps of these processescan be accomplished by one or more different systems, devices orcomponents. In some embodiments, some of the steps could be accomplishedby a response system 188 of a motor vehicle. In some cases, some of thesteps can be accomplished by the ECU 106 of a motor vehicle 100. Inother embodiments, some of the steps could be accomplished by othercomponents of a motor vehicle, including but not limited to, the vehiclesystems 126. For each process discussed below and illustrated in theFigures it will be understood that in some embodiments one or more ofthe steps could be optional. Additionally, it will be appreciated thateach system and method discussed below is applicable to embodiments thatdetermine one or more driver states or combine driver states as will bediscussed in further detail herein.

FIG. 24A illustrates an embodiment of a process for controlling one ormore vehicle systems in a motor vehicle depending on the state of thedriver. In some embodiments, some of the following steps could beaccomplished by a response system 188 of a motor vehicle. In some cases,some of the following steps can be accomplished by an ECU 106 of a motorvehicle. In other embodiments, some of the following steps could beaccomplished by other components of a motor vehicle, such as vehiclesystems 126. In still other embodiments, some of the following stepscould be accomplished by any combination of systems or components of thevehicle. It will be understood that in some embodiments one or more ofthe following steps can be optional. For purposes of reference, thefollowing method discusses components shown in FIGS. 1A, 1B through 3,including the response system 188.

In step 2402, the response system 188 can receive monitoringinformation. In some cases, the monitoring information can be receivedfrom one or more sensors. In other cases, the monitoring information canbe received from one or more monitoring systems. In still other cases,the monitoring information can be received from one or more vehiclesystems. In still other cases, the monitoring information can bereceived from any other device of the motor vehicle 100. In still othercases, the monitoring information can be received from any combinationof sensors, monitoring systems (e.g., the monitoring systems 300),vehicles systems, or other devices. For example, and as discussed above,the monitoring information can be received from physiological monitoringsystems and sensors, behavioral monitoring systems and sensors,vehicular monitoring systems and sensors, identification systems andsensors, or any combination thereof.

In step 2404, the response system 188 can determine the driver state. Insome cases, the driver state can be normal or drowsy. In other cases,the driver state can range over three or more states ranging betweennormal and very drowsy (or even asleep). In still other cases, thedriver state can be normal or distracted. In other cases, the driverstate can be alert, normal, distracted, or drowsy. In other cases, thedriver state can range over three or more states ranging between normaland very distracted. In this step, the response system 188 can use anyinformation received during step 2402, including information from anykinds of sensors or systems. For example, in one embodiment, responsesystem 188 can receive information from an optical sensing device thatindicates the driver has closed his or her eyes for a substantial periodof time. In another embodiment, response system 188 can receiveinformation from an optical sensing device that indicates the driver isnot looking forward. Other examples of determining the state of a driverare discussed in detail below.

In step 2406, the response system 188 can determine whether the driveris distracted or other diminished state, for example drowsy. If thedriver is not distracted, the response system 188 can proceed back tostep 2402 to receive additional monitoring information. If, however, thedriver is distracted, the response system 188 can proceed to step 2408.In step 2408, the response system 188 can automatically modify thecontrol of one or more vehicle systems, including any of the vehiclesystems discussed above. By automatically modifying the control of oneor more vehicle systems, the response system 188 can help to avoidvarious hazardous situations that can be caused by a drowsy and/ordistracted driver.

As discussed above, at step 2408, if the driver is distracted theresponse system 188 can automatically modify the control of one or morevehicle systems, including any of the vehicle systems discussed above.However, in some embodiments, a user may not want any vehicle systemsmodified or adjusted. In these cases, the user can switch a user inputdevice 152, or a similar kind of input device, to the OFF position. Thiscould have the effect of turning off all driver state monitoring andwould further prevent the response system 188 from modifying the controlof any vehicle systems. Moreover, the response system 188 could bereactivated at any time by switching user input device 152 to the ONposition. In other embodiments, additional switches or buttons could beprovided to turn on/off individual monitoring systems.

In a further embodiment, the response system 188 can automaticallyoverride, cancel, or turn OFF the modification or adjustment of one ormore vehicle systems based on the driver state. For example, if at step2406 it is determined the driver state is not distracted (e.g., alert,vigilant), the response system 188 can automatically turn off all driverstate monitoring and prevent the response system 188 from modifying thecontrol of any vehicle systems. The response system 188 canautomatically reactivate the driver state monitoring upon detecting adriver state that is distracted (e.g., not alert, not vigilant, drowsy).In another embodiment, the response system 188 can automaticallyoverride and/or cancel the modification or adjustment of one or morevehicle systems based on the driver state and information from one ormore vehicle systems 126 (e.g., a vehicular state). As an illustrativeexample, if the driver state is vigilant (e.g., alert, not drowsy) andthe blind spot indicator system 224 indicates a target vehicle is notpresent in a blind spot monitoring zone, the response system 188 canturn off warnings and modifications from the lane departure warningsystem 222 for lane departures toward said blind spot monitoring zone.These embodiments will be described in more detail herein.

FIG. 24B illustrates an embodiment of a process for controlling one ormore vehicle systems in a motor vehicle depending on the state of thedriver similar to FIG. 24A but with identification of a driver. In someembodiments, some of the following steps could be accomplished by aresponse system 188 of a motor vehicle. In some cases, some of thefollowing steps can be accomplished by an ECU 106 of a motor vehicle. Inother embodiments, some of the following steps could be accomplished byother components of a motor vehicle, such as vehicle systems 126. Instill other embodiments, some of the following steps could beaccomplished by any combination of systems or components of the vehicle.It will be understood that in some embodiments one or more of thefollowing steps can be optional. For purposes of reference, thefollowing method discusses components shown in FIGS. 1A, 1B through 3,including the response system 188.

In step 2410, the response system 188 can receive monitoringinformation. In some cases, the monitoring information can be receivedfrom one or more sensors. In other cases, the monitoring information canbe received from one or more monitoring systems. In still other cases,the monitoring information can be received from one or more vehiclesystems. In still other cases, the monitoring information can bereceived from any other device of the motor vehicle 100. In still othercases, the monitoring information can be received from any combinationof sensors, monitoring systems (e.g., the monitoring systems 300),vehicles systems, or other devices. For example, and as discussed above,the monitoring information can be received from physiological monitoringsystems and sensors, behavioral monitoring systems and sensors,vehicular monitoring systems and sensors, identification systems andsensors, or any combination thereof.

In step 2412, the response system 188 can determine the driver state. Insome cases, the driver state can be normal or drowsy. In other cases,the driver state can range over three or more states ranging betweennormal and very drowsy (or even asleep). In still other cases, thedriver state can be normal or distracted. In other cases, the driverstate can be alert, normal, distracted, or drowsy. In other cases, thedriver state can range over three or more states ranging between normaland very distracted. In this step, the response system 188 can use anyinformation received during step 2410, including information from anykinds of sensors or systems. For example, in one embodiment, responsesystem 188 can receive information from an optical sensing device thatindicates the driver has closed his or her eyes for a substantial periodof time. In another embodiment, response system 188 can receiveinformation from an optical sensing device that indicates the driver isnot looking forward. Other examples of determining the state of a driverare discussed in detail below.

In the embodiment shown in FIG. 24B, determining the driver state atstep 2412 can include identifying the driver at step 2414. Any of thesystems and methods described above to identify the driver in SectionIII (B) (4) can be used for personal identification of the driver. Insome embodiments, the monitoring information received at step 2410 canbe used at step 2414 to identify the driver. As discussed above, onceidentification of the driver is determined, the identified driver can beassociated with a user (e.g., driver) profile including parameters,data, and data (e.g., monitoring information) tracked overtime specificto the driver. The ECU 106 can store the user profile (not shown) at thememory 110 and/or the disk 112 shown in FIG. 1A.

Accordingly, the data stored in the user profile can provide normativeand baseline data of the driver, which can be used to determine a driverstate. More specifically, at step 2416, determining the driver state caninclude comparing stored information (e.g., the stored data/monitoringinformation) in the user profile with the monitoring informationreceived at step 2410. In some embodiments, the stored information andthe monitoring information compared at step 2416 can both be associatedwith the same parameter, type of monitoring information, and/or vehiclesystem.

As an illustrative example, at step 2410, the response system 188 canreceive steering information from the electronic power steering system132 and/or the touch steering wheel system 134. The steering informationcan include a steering input signal, which may indicate whether thedriver's steering is smooth, erratic, and/or jerky. The steeringinformation may also include the hand position of the driver. Forexample, the response system 188 can determine whether the driver haszero, one or two hands on the steering wheel. The steering informationreceived at step 2410 can be compared to stored steering informationretrieved by the response system 188 from the user profile. The storedsteering information can indicate a normative and/or a baseline steeringinput signal. The stored steering information can also indicate how manyhands the user has in contact with the steering wheel 134 when theinformation is stored. In other words, the response system 188 can storea normative and/or baseline steering input signal for the driver whenthe driver is using one hand and also when the driver is using twohands. Accordingly, if the stored steering information is a steeringinput signal that indicates, while using one hand, the driver's steeringis normally smooth and the steering information received at step 2410indicates, while using one hand, the driver's steering is erratic, thedriver state may be determined to be distracted at step 2412. Saiddifferently, if the stored steering information is inconsistent with thesteering information received at step 2410, the driver state may bedetermined to be distracted at step 2412. This can also apply tosteering input signals where the driver is using two hands on thesteering wheel 134. In addition, if the driver's stored steeringinformation indicates that the driver's one-handed use of the steeringwheel is smoother and less erratic than the driver's two-handed use ofthe steering wheel, then the system can adjust any vehicle systemmodifications discussed in Section VI accordingly.

At step 2418, the response system 188 can determine whether the driveris distracted or other diminished state, for example drowsy. If thedriver is not distracted, the response system 188 can proceed back tostep 2410 to receive additional monitoring information. If, however, thedriver is distracted, the response system 188 can proceed to step 2420.In step 2420, the response system 188 can automatically modify thecontrol of one or more vehicle systems, including any of the vehiclesystems discussed above. By automatically modifying the control of oneor more vehicle systems, the response system 188 can help to avoidvarious hazardous situations that can be caused by a drowsy and/ordistracted driver.

As discussed above, at step 2420, if the driver is distracted theresponse system 188 can automatically modify the control of one or morevehicle systems, including any of the vehicle systems discussed above.Referring to the illustrative example discussed above, if it isdetermined the driver is distracted based on the comparison of steeringinformation, the response system 188 can modify the electronic powersteering system 132 to provide more assistance based on the driverstate.

FIG. 25 is a table emphasizing the response system 188 impact on variousvehicle systems due to changes in the driver's state, as well as thebenefits to the driver for each change according to one embodiment. Inparticular, column 2502 lists the various vehicle systems, which includemany of the vehicle systems 126 discussed above and shown in FIG. 2.Column 2504 describes how response system 188 affects the operation ofeach vehicle system when the driver's state is such that the driver canbe distracted, drowsy, less attentive, and/or impaired. Column 2506describes the benefits for the response system impacts described incolumn 2504. Column 2508 describes the type of impact performed byresponse system 188 for each vehicle system. In particular, in column2508 the impact of response system 188 on each vehicle system isdescribed as either “control” type or “warning” type. The control typeindicates that the operation of a vehicle system is modified by thecontrol system. The warning type indicates that the vehicle system isused to warn or otherwise alert a driver.

As indicated in FIG. 25, upon detecting that a driver is drowsy orotherwise inattentive, the response system 188 can control theelectronic stability control system 202, the antilock brake system 204,the brake assist system 206, and the brake prefill system 208 in amanner that compensates for the potentially slower reaction time of thedriver. For example, in some cases, response system 188 can operate theelectronic stability control system 202 to improve steering precisionand enhance stability. In some cases, response system 188 can operatethe antilock brake system 204 so that the stopping distance isdecreased. In some cases, response system 188 can control the brakeassist system 206 so that an assisted braking force is applied sooner.In some cases, response system 188 can control the brake prefill system208 so the brake lines are automatically prefilled with brake fluid whena driver is drowsy. These actions can help to improve the steeringprecision and brake responsiveness when a driver is drowsy.

Additionally, upon detecting that a driver is distracted, drowsy orotherwise inattentive, the response system 188 can control the low speedfollow system 212, the cruise control system 214, the automatic cruisecontrol system 216, the collision warning system 218, the collisionmitigation braking system 220, the lane departure warning system 222,the blind spot indicator system 224 and the lane keep assist system 226to provide protection due to the driver's lapse of attention. Forexample, the low speed follow system 212, the cruise control system 214,and the lane keep assist system 226 could be disabled when the driver isdistracted and/or drowsy to prevent unintended use of these systems.Likewise, the collision warning system 218, the collision mitigationbraking system 220, the lane departure warning system 222, and the blindspot indicator system 224 could warn a driver sooner about possiblepotential hazards. In some cases, the automatic cruise control system216 could be configured to increase the minimum gap distance between themotor vehicle 100 and the preceding vehicle.

In some embodiments, upon detecting that a driver is drowsy or otherwiseinattentive, the response system 188 can control the electronic powersteering system 132, the visual devices 140, the audio devices 144, thetactile devices 148, the climate control system 234 (such as HVAC), andthe electronic pretensioning system 236 for a seat belt to supplementthe driver's alertness. For example, the electronic power steeringsystem 132 can be controlled to decrease power steering assistance. Thisrequires the driver to apply more effort and can help improve awarenessor alertness. The visual devices 140 and the audio devices 144 can beused to provide visual feedback and audible feedback, respectively. Thetactile devices 148 and the electronic pretensioning system 236 can beused to provide tactile feedback to a driver. In addition, the climatecontrol system 234 can be used to change the cabin or driver temperatureto effect the drowsiness of the driver. For example, by changing thecabin temperature the driver can be made more alert.

The various systems listed in FIG. 25 are only intended to be exemplaryand other embodiments could include additional vehicle systems that canbe controlled by the response system 188. Moreover, these systems arenot limited to a single impact or function. In addition, these systemsare not limited to a single benefit. Instead, the impacts and benefitslisted for each system are intended as examples. A detailed explanationof the control of many different vehicle systems is discussed in detailbelow and shown in the Figures.

A response system can include provisions for determining a level ofdrowsiness for a driver and/or a level of distraction for a driver. Theterm “level of drowsiness” as used throughout this detailed descriptionand in the claims refers to any numerical or other kind of value fordistinguishing between two or more states of drowsiness. For example, insome cases, the level of drowsiness can be given as a percentage between0% and 100%, where 0% refers to a driver that is totally alert and 100%refers to a driver that is fully drowsy or even asleep. In other cases,the level of drowsiness could be a value in the range between 1 and 10.In still other cases, the level of drowsiness is not a numerical value,but could be associated with a given discrete state, such as “notdrowsy,” “slightly drowsy,” “drowsy,” “very drowsy” and “extremelydrowsy.” Moreover, the level of drowsiness could be a discrete value ora continuous value. In some cases, the level of drowsiness can beassociated with a driver state index, which is discussed in furtherdetail below.

The term “level of distraction” as used throughout this detaileddescription and in the claims refers to any numerical or other kind ofvalue for distinguishing between two or more states of distraction. Forexample, in some cases, the level of distraction can be given as apercentage between 0% and 100%, where 0% refers to a driver that istotally attentive and 100% refers to a driver that is fully distracted.In other cases, the level of distraction could be a value in the rangebetween 1 and 10. In still other cases, the level of distraction is nota numerical value, but could be associated with a given discrete state,such as “not distracted,” “slightly distracted,” “distracted”, “verydistracted” and “extremely distracted”. Moreover, the level ofdistraction could be a discrete value or a continuous value. In somecases, the level of distraction can be associated with a driver stateindex, which is discussed in further detail below. In further cases, thelevel of distraction can indicate the driver is engaged in a secondarytask (e.g., other than the primary task of driving).

FIG. 26 illustrates an embodiment of a process of modifying theoperation of a vehicle system according to the level of distractiondetected. In some embodiments, some of the following steps could beaccomplished by a response system 188 of a motor vehicle. In some cases,some of the following steps can be accomplished by an ECU 106 of a motorvehicle. In other embodiments, some of the following steps could beaccomplished by other components of a motor vehicle, such as vehiclesystems 126. In still other embodiments, some of the following stepscould be accomplished by any combination of systems or components of thevehicle. It will be understood that in some embodiments one or more ofthe following steps can be optional. For purposes of reference, thefollowing method discusses components shown in FIGS. 1A, 1B through 3,including the response system 188.

In step 2602, response system 188 can receive monitoring information. Insome cases, the monitoring information can be received from one or moresensors. In other cases, the monitoring information can be received fromone or more autonomic monitoring systems. In still other cases, themonitoring information can be received from one or more vehicle systems.In still other cases, the monitoring information can be received fromany other device of the motor vehicle 100. In still other cases, themonitoring information can be received from any combination of sensors,monitoring systems, vehicles systems or other devices. For example, andas discussed above, the monitoring information can be received fromphysiological monitoring systems and sensors, behavioral monitoringsystems and sensors, vehicular monitoring systems and sensors,identification systems and sensors, or any combination thereof.

In step 2604, the response system 188 can determine if the driver isdistracted (e.g., not alert, drowsy). If the driver is not distracted,the response system 188 can return back to step 2602. If the driver isdistracted, the response system 188 can proceed to step 2606. In step2606, the response system 188 can determine the level of distraction(e.g., drowsiness). As discussed above, the level of distraction couldbe represented by a numerical value or could be a discrete state labeledby a name or variable. In step 2608, the response system 188 can modifythe control of one or more vehicle systems according to the level ofdistraction.

Examples of systems that can be modified according to the level ofdistraction include, but are not limited to: the electronic stabilitycontrol system 202, the antilock brake system 204, the brake assistsystem 206, the brake prefill system 208, the EPB system 210, the lowspeed follow system 212, the automatic cruise control system 216, thecollision warning system 218, the lane keep assist system 226, the blindspot indicator system 224, the climate control system 234, and theelectronic pretensioning system 236. In addition, the electronic powersteering system 132 could be modified according to the level ofdistraction, as could the visual devices 140, the audio devices 144, andthe tactile devices 148. In some embodiments, the timing and/orintensity associated with various warning indicators (visual indicators,audible indicators, haptic indicators, etc.) could be modified accordingto the level of distraction. For example, in one embodiment, theelectronic pretensioning system 236 could increase or decrease theintensity and/or frequency of automatic seat belt tightening to warn thedriver at a level appropriate for the level of distraction.

As an example, when a driver is extremely distracted (e.g., extremelydrowsy), the antilock brake system 204 can be modified to achieve ashorter stopping distance than when a driver is somewhat distracted. Thelevel of brake assistance provided by the brake assist system 206 couldbe varied according to the level of drowsiness, with assistanceincreased with distraction. As another example, the brake prefill system208 could adjust the amount of brake fluid delivered during a prefill orthe timing of the prefill according to the level of distraction. Inaddition, the headway distance for the automatic cruise control system216 could be increased with the level of distraction. In addition, theerror between the yaw rate and the steering yaw rate determined byelectronic stability control system 202 could be decreased in proportionto the level of distraction. In some cases, the collision warning system218 and the lane departure warning system 222 could provide earlierwarnings to a distracted driver, where the timing of the warnings ismodified in proportion to the level of distraction. Likewise, thedetection area size associated with the blind spot indicator system 224could be varied according to the level of distraction. In some cases,the strength of a warning pulse generated by the electronicpretensioning system 236 can vary in proportion to the level ofdrowsiness.

In addition, the climate control system 234 can vary the number ofdegrees that the temperature is changed according to the level ofdistraction. Moreover, the brightness of the lights activated by thevisual devices 140 when a driver is distracted could be varied inproportion to the level of distraction. In addition, the volume of soundgenerated by the audio devices 144 could be varied in proportion to thelevel of distraction. In addition, the amount of vibration or tactilestimulation delivered by the tactile devices 148 could be varied inproportion to the level of distraction. In some cases, the maximum speedat which the low speed follow system 212 operates could be modifiedaccording to the level of distraction. Likewise, the ON/OFF setting orthe maximum speed at which the cruise control system 214 can be set canbe modified in proportion to the level of distraction. Additionally, thedegree of power steering assistance provided by the electronic powersteering system 132 could be varied in proportion to the level ofdistraction. In addition, the distance that the collision mitigationbraking system 220 begins to brake can be lengthened or the lane keepassist system 226 could be modified so that the driver must provide moreinput to the system.

FIG. 27 illustrates another embodiment of a process of modifying theoperation of a vehicle system according to the level of drowsinessdetected. In some embodiments, some of the following steps could beaccomplished by a response system 188 of a motor vehicle. In some cases,some of the following steps can be accomplished by an ECU 106 of a motorvehicle. In other embodiments, some of the following steps could beaccomplished by other components of a motor vehicle, such as vehiclesystems 126. In still other embodiments, some of the following stepscould be accomplished by any combination of systems or components of thevehicle. It will be understood that in some embodiments one or more ofthe following steps can be optional. For purposes of reference, thefollowing method discusses components shown in FIGS. 1A, 1B through 3,the including response system 188.

In step 2702, the response system 188 can receive monitoringinformation, as discussed above and with respect to step 2602 of FIG.26. In step 2704, the response system 188 can receive any kind ofvehicle operating information from one or more vehicle systems. The typeof operating information received during step 2704 can vary according tothe type of vehicle system involved. For example, if the current processis used for operating a brake assist system, the operating informationreceived can be brake pressure, vehicle speed and other operatingparameters related to a brake assist system. As another example, if thecurrent process is used for operating an electronic stability controlsystem, the operation information can include yaw rate, wheel speedinformation, steering angle, lateral G, longitudinal G, road frictioninformation as well as any other information used for operating anelectronic stability control system.

Next, in step 2706, the response system 188 can determine a driver stateindex of the driver. The term “driver state index” refers to a measureof the drowsiness and/or distractedness of a driver. In some cases, thedriver state index could be given as a numerical value. In other cases,the driver state index could be given as a non-numerical value.Moreover, the driver state index can range from values associated withcomplete alertness to values associated with extreme drowsiness or evena state in which the driver is asleep. In one embodiment, the driverstate index could take on the values 1, 2, 3 and 4, where 1 is the leastdrowsy and 4 is the most drowsy. In another embodiment, the driver stateindex could take on values from 1-10. Further, the driver state indexcan range from values associated with complete attentiveness to valuesassociated with extreme distraction. In one embodiment, the driver stateindex could take on the values 1, 2, 3 and 4, where 1 is the leastdistracted and 4 is the most distracted.

In step 2708, the response system 188 can determine a control parameter.The term “control parameter” as used throughout this detaileddescription and in the claims refers to a parameter used by one or morevehicle systems. In some cases, a control parameter can be an operatingparameter that is used to determine if a particular function should beactivated for a given vehicle system. For example, in situations wherean electronic stability control system is used, the control parametercan be a threshold error in the steering yaw rate that is used todetermine if stability control should be activated. As another example,in situations where automatic cruise control is used, the controlparameter can be a parameter used to determine if cruise control shouldbe automatically turned off. Further examples of control parameters arediscussed in detail below and include, but are not limited to: stabilitycontrol activation thresholds, brake assist activation thresholds, blindspot monitoring zone thresholds, time to collision thresholds, roadcrossing thresholds, lane keep assist system status, low speed followstatus, electronic power steering status, automatic cruise controlstatus as well as other control parameters.

FIGS. 28 and 29 illustrate schematic views of a general method fordetermining a control parameter using the driver state index of thedriver as well as vehicle operating information. In particular, FIG. 28illustrates a schematic view of how the driver state index can be usedto retrieve a control coefficient. A control coefficient can be anyvalue used in determining a control parameter. In some cases, thecontrol coefficient varies as a function of driver state index and isused as an input for calculating the control parameter. Examples ofcontrol coefficients include, but are not limited to electronicstability control system coefficients, brake assist coefficients, blindspot zone warning coefficients, warning intensity coefficients, forwardcollision warning coefficients, lane departure warning coefficients andlane keep assist coefficients. Some systems cannot use a controlcoefficient to determine the control parameter. For example, in somecases, the control parameter can be determined directly from the driverstate index.

In one embodiment, the value of the control coefficient 2802 increasesfrom 0% to 25% as the driver state index increases from 1 to 4. In somecases, the control coefficient can serve as a multiplicative factor forincreasing or decreasing the value of a control parameter. For example,in some cases when the driver state index is 4, the control coefficientcan be used to increase the value of a control parameter by 25%. Inother embodiments, the control coefficient could vary in any othermanner. In some cases, the control coefficient could vary linearly as afunction of driver state index. In other cases, the control coefficientcould vary in a nonlinear manner as a function of driver state index. Instill other cases, the control coefficient could vary between two ormore discrete values as a function of driver state index.

FIG. 29 illustrates a calculation unit 2902 for determining a controlparameter. The calculation unit 2902 receives a control coefficient 2904and vehicle operating information 2906 as inputs. The calculation unit2902 outputs the control parameter 2908. The vehicle operatinginformation 2906 can include any information necessary to calculate acontrol parameter. For example, in situations where the vehicle systemis an electronic stability control system, the system can receive wheelspeed information, steering angle information, roadway frictioninformation, as well as other information necessary to calculate acontrol parameter that is used to determine when stability controlshould be activated. Moreover, as discussed above, the controlcoefficient 2904 can be determined from the driver state index using,for example, a look-up table. The calculation unit 2902 then considersboth the vehicle operating information 2906 and the control coefficient2904 in calculating the control parameter 2908.

It will be understood that the calculation unit 2902 is intended to beany general algorithm or process used to determine one or more controlparameters. In some cases, the calculation unit 2902 can be associatedwith the response system 188 and/or the ECU 106. In other cases,however, the calculation unit 2902 could be associated with any othersystem or device of the motor vehicle 100, including any of the vehiclesystems discussed previously.

In some embodiments, a control parameter can be associated with a statusor state of a given vehicle system. FIG. 30 illustrates an embodiment ofa general relationship between the driver state index of the driver anda system status 3002. The system shown here is general and could beassociated with any vehicle system. For low driver state index (1 or 2),the system status 3002 is ON. However, if the driver state indexincreases to 3 or 4 the system status 3002 is turned OFF. In still otherembodiments, a control parameter could be set to multiple different“states” according to the driver state index. Using this arrangement,the state of a vehicle system can be modified according the driver stateindex of a driver.

Generally, the driver state index can be determined using any of themethods discussed throughout this detailed description for detectingdriver state as it relates to distraction and/or drowsiness. Inparticular, the level of drowsiness and/or level of distraction can bedetected by sensing different degrees of driver state. For example, asdiscussed below, drowsiness and/or distraction in a driver can bedetected by sensing eyelid movement and/or head movement. In some cases,the degree of eyelid movement (the degree to which the eyes are open orclosed) or the degree of head movement (how tilted the head is) could beused to determine the driver state index. In other cases, the monitoringsystems 300 could be used to determine the driver state index. In stillother cases, the vehicle systems could be used to determine the driverstate index. For example, the degree of unusual steering behavior or thedegree of lane departures, alone or in combination, can indicate acertain driver state index.

A. Types of Driver States

As discussed above, a motor vehicle can include provisions for assessingthe state of a driver and automatically adjusting the operation of oneor more vehicle systems in response to one or more driver states. InSection I, a “driver state” is defined in detail and can refer to ameasurement of a state of the biological being and/or a state of theenvironment of the biological being (e.g., a vehicle). The followingdescription discusses specific driver states based on specific types ofmonitoring systems and/or monitoring information, namely, aphysiological driver state, a behavioral driver state and avehicular-sensed driver state.

1. Physiological Driver State

A physiological driver state is based on physiological information fromphysiological monitoring systems and sensors, as discussed above insection III (B) (2). Physiological information includes informationabout the human body (e.g., a driver) derived intrinsically. Saiddifferently, physiological information is measured by medical means andquantifies an internal characteristic of a human body. Physiologicalinformation is typically not externally observable to the human eye.However, in some cases, physiological information is observable byoptical means, for example, heart rate measured by an optical device.Physiological information can include, but is not limited to, heartrate, blood pressure, oxygen content, blood alcohol content, respiratoryrate, perspiration rate, skin conductance, brain wave activity,digestion information, salivation information, among others.Physiological information can also include information about theautonomic nervous systems of the human body derived intrinsically.

The following examples describe a variety of different methods fordetermining a physiological driver state, for example a physiologicaldriver state based on respiratory rate information and autonomicinformation. It is understood that the methods for determining aphysiological driver state can also include physiological driver statesbased on other types of physiological information.

FIG. 31 illustrates a schematic view of an embodiment of the motorvehicle 100, in which the response system 188 is capable of detectingrespiratory rate information (e.g., physiological information). Inparticular, using a bio-monitoring sensor 180, the ECU 106 can determinethe number of breaths per minute taken by driver 102. In one embodiment,the response system 188 can receive respiratory rate information fromrespiratory monitoring system 312. The respiratory rate information canbe analyzed to determine if the measured breaths per minute coincideswith a normal state or a distracted (e.g., drowsy) state. Breaths perminute is given as an example.

Although FIG. 31 schematically describes detecting respiratory rateinformation to determine a physiological driver state, it is understoodthat other types of physiological information can be monitored and usedto determine one or more physiological driver states. For example, theresponse system 188 can detect and/or receive heart rate informationfrom a heart rate monitoring system 302. As discussed above, the heartrate monitoring system 302 can include heart rate sensors 304, bloodpressure sensors 306, oxygen content sensors 308 and blood alcoholcontent sensors 310, as well as any other kinds of sensors for detectingheart information and/or cardiovascular information. These sensors couldbe disposed in a dashboard, steering wheel (e.g., touch steering wheelsystem 134), seat, seat belt, armrest or other component to detect theheart information of a driver.

The heart information and/or cardiovascular information can be analyzedto determine a physiological driver state. For example, the heartinformation can be analyzed to determine if a heart rate (e.g., beatsper minute) coincides with a particular physiological driver state. Forexample, a high heart rate can coincide with a stressed driver state. Alow heart rate can coincide with a drowsy driver state. In one example,a physiological driver state and changes in a physiological driver statecan be based on parasympathetic and sympathetic activity levels byanalyzing heart rate information as discussed in in U.S. Pat. No.9,420,958, entitled System and Method for Determining Changes in a BodyState, which is incorporated by reference in its entirety herein.

In another embodiment, the ECU 106 can determine the blood pressure ofthe driver from information received by the blood pressure sensors 306.The blood pressure can be analyzed to determine if the blood pressurecoincides with a particular physiological driver state. For example, ahigh blood pressure level can coincide with a stressed driver state. Ina further embodiment, the ECU 106 can determine the blood oxygen contentof the driver based on information received by the oxygen contentsensors 308. The blood oxygen content can be analyzed to determine ifthe blood oxygen content coincides with a particular physiologicaldriver state. For example, low blood oxygen levels can coincide with adrowsy driver state.

In another embodiment, the ECU 106 can determine blood alcohol content(BAC) (e.g., blood alcohol levels) of the driver from informationreceived by the blood alcohol content sensors 310. For example, anoptical sensor can emit light towards the driver's skin and measure atissue alcohol concentration based on the amount of light that isreflected back by the skin. The BAC can be analyzed to determine if theBAC coincides with a particular physiological driver state. For example,high BAC can coincide with an impaired/distracted driver state (e.g., anintoxicated driver).

In some embodiments, the response system 188 can detect and/or receiveperspiration information from a perspiration monitoring system 314. Theperspiration monitoring system 314 can include any devices or systemsfor sensing perspiration or sweat from a driver. Accordingly, the ECU106 can determine a level of perspiration from the driver to determineif the perspiration coincides with a particular physiological driverstate. For example, if the driver's perspiration rate is high, this cancoincide with a stressed driver state.

In some embodiments, the response system 188 can detect and/or receivepupil dilation information from a pupil dilation monitoring system 316for sensing the amount of pupil dilation, or pupil size, in a driver.Accordingly, the ECU 106 can analyze the pupil size to determineparticular physiological driver state. For example, enlarged (e.g.,dilated pupils) can coincide with a drowsy or stressed driver state.

Additionally, in some embodiments, the response system 188 can detectand/or receive brain information from a brain monitoring system 318. Insome cases, the brain monitoring system 318 could includeelectroencephalogram (EEG) sensors 320, functional near infraredspectroscopy (fNIRS) sensors 322, functional magnetic resonance imaging(fMRI) sensors 324 as well as other kinds of sensors capable ofdetecting brain information. Such sensors could be located in anyportion of the motor vehicle 100. In some cases, sensors associated withthe brain monitoring system 318 could be disposed in a headrest. Inother cases, sensors could be disposed in the roof of the motor vehicle100. In still other cases, sensors could be disposed in any otherlocations. Accordingly, the ECU 106 can analyze the brain information todetermine a particular physiological driver state. For example, abnormalbrain waves can coincide with a health state, for example, a seizure.

In some embodiments, the response system 188 can detect and/or receivedigestion information from a digestion monitoring system 326. In otherembodiments, the response system 188 can detect and/or receivesalivation information from a salivation monitoring system 328. In somecases, monitoring digestion and/or salivation could also help indetermining a physiological driver state. For example, the ECU 106 cananalyze the digestion information to determine that the body isdigesting food and blood is being directed toward the stomach that canlead to a drowsy driver state. In another example, if the ECU 106determines the body is poorly digesting food, the ECU 106 can determinethe driver is in an inattentive or drowsy driver state.

Referring now to FIG. 32, an embodiment of a process for detectingdistraction (e.g., drowsiness) by monitoring the physiologicalinformation (e.g., autonomic information) of a driver is shown. In someembodiments, some of the following steps could be accomplished by aresponse system 188 of a motor vehicle. In some cases, some of thefollowing steps can be accomplished by an ECU 106 of a motor vehicle. Inother embodiments, some of the following steps could be accomplished byother components of a motor vehicle, such as the vehicle systems 126. Instill other embodiments, some of the following steps could beaccomplished by any combination of systems or components of the vehicle.It will be understood that in some embodiments one or more of thefollowing steps can be optional. For purposes of reference, thefollowing method discusses components shown in FIGS. 1A, 1B through 3,including the response system 188.

In step 3202, the response system 188 can receive physiologicalinformation related to the autonomic nervous system of the driver. Insome cases, the information can be received from a sensor. The sensorcould be associated with any portion of the motor vehicle 100 includinga seat, armrest, or any other portion. Moreover, the sensor could be aportable sensor in some cases. Furthermore, the physiologicalinformation could be received from any physiological monitoring systemsand/or sensors described in Section III (B) (1).

In step 3204, the response system 188 can analyze the autonomicinformation. Generally, any method of analyzing autonomic information todetermine if a driver is drowsy could be used. It will be understoodthat the method of analyzing the autonomic information can varyaccording to the type of autonomic information being analyzed. In step3206, the response system 188 can determine the driver state index(e.g., a physiological driver state index) of the driver based on theanalysis conducted during step 3204. In some embodiments discussedherein, one or more vehicle systems can be modified based on the driverstate index determined at step 3206.

2. Behavioral Driver State

A behavioral driver state is based on behavioral information frombehavioral monitoring systems and sensors, as discussed above in sectionIII (B) (3). Behavioral information includes information about the humanbody derived extrinsically. Behavioral information is typicallyobservable externally to the human eye. For example, behavioralinformation can include eye movements, mouth movements, facialmovements, facial recognition, head movements, body movements, handpostures, hand placement, body posture, gesture recognition, amongothers. The following examples describe a variety of different methodsfor determining a behavioral driver state, for example a behavioraldriver state based on eye movement, head movement and head position. Itis understood that the methods for operating vehicle systems in responseto a behavioral driver state can also include behavioral driver statesbased on other types of behavioral information.

As discussed above, a response system can include provisions fordetecting the state of a driver, for example a behavioral state of adriver. In one example, the response system can detect the state of adriver by monitoring the eyes of a driver. FIG. 33 illustrates aschematic view of a scenario in which the response system 188 is capableof monitoring the state or behavior of a driver. Referring to FIG. 33,the ECU 106 can receive information from an optical sensing device 162.In some cases, the optical sensing device 162 can be a video camera thatis mounted in the dashboard of the motor vehicle 100. The informationcan comprise a sequence of images 3300 that can be analyzed to determinethe state of driver 102. A first image 3302 shows a driver 102 in afully awake (e.g., attentive) state, with eyes 3304 wide open. However,a second image 3306 shows the driver 102 in a drowsy (e.g., distracted)state, with eyes 3304 half open. Finally, a third image 3308 shows thedriver 102 in a very drowsy (distracted) state with eyes 3304 fullyclosed. In some embodiments, the response system 188 can be configuredto analyze various images of the driver 102. More specifically, theresponse system 188 can analyze the movement of eyes 3304 to determineif a driver is in a normal state or a drowsy (e.g., distracted) state.

It will be understood that any type of algorithm known in the art foranalyzing eye movement from images can be used. In particular, any typeof algorithm that can recognize the eyes and determine the position ofthe eyelids between a closed and open position can be used. Examples ofsuch algorithms can include various pattern recognition algorithms knownin the art.

In other embodiments, a thermal sensing device 166 can be used to senseeyelid movement. For example, as the eyelids move between opened andclosed positions, the amount of thermal radiation received at a thermalsensing device 166 can vary. In other words, the thermal sensing device166 can be configured to distinguish between various eyelid positionsbased on variations in the detected temperature of the eyes.

FIG. 34 illustrates an embodiment of a process for detecting drowsinessby monitoring eye movement in the driver. In some embodiments, some ofthe following steps could be accomplished by a response system 188 of amotor vehicle. In some cases, some of the following steps can beaccomplished by an ECU 106 of a motor vehicle. In other embodiments,some of the following steps could be accomplished by other components ofa motor vehicle, such as vehicle systems 126. In still otherembodiments, some of the following steps could be accomplished by anycombination of systems or components of the vehicle. It will beunderstood that in some embodiments one or more of the following stepscan be optional. For purposes of reference, the following methoddiscusses components shown in FIGS. 1A, 1B through 3, including theresponse system 188.

In step 3402, the response system 188 can receive optical/thermalinformation. In some cases, optical information could be received from acamera or from an optical sensing device 162. In other cases, thermalinformation could be received from a thermal sensing device 166. Instill other cases, both optical and thermal information could bereceived from a combination of optical and thermal devices.

In step 3404, the response system 188 can analyze eyelid movement. Bydetecting eyelid movement, the response system 188 can determine if theeyes of a driver are open, closed or in a partially closed position. Theeyelid movement can be determined using either optical information orthermal information received during step 3402. Moreover, as discussedabove, any type of software or algorithm can be used to determine eyelidmovement from the optical or thermal information. Although the currentembodiment comprises a step of analyzing eyelid movement, in otherembodiments the movement of the eyeballs could also be analyzed.

In step 3406, the response system 188 determines the driver state index(e.g., the behavioral driver state index) of the driver according to theeyelid movement. The driver state index can have any value. In somecases, the value ranges between 1 and 4, with 1 being the least drowsyand 4 being the drowsiest state. In some cases, the value ranges between1 and 4, with 1 being the least distracted and 4 being the mostdistracted state. In some cases, to determine the driver state index theresponse system 188 determines if the eyes are closed or partiallyclosed for extended periods. In order to distinguish drooping eyelidsdue to drowsiness (e.g., distraction) from blinking, the response system188 can use a threshold time that the eyelids are closed or partiallyclosed. If the eyes of the driver are closed or partially closed forperiods longer than the threshold time, the response system 188 candetermine that this is due to drowsiness (e.g., distraction). In suchcases, the driver can be assigned a driver state index that is greaterthan 1 to indicate that the driver is drowsy (e.g., distracted).Moreover, the response system 188 can assign different driver stateindex values for different degrees of eyelid movement or eyelid closure.

In some embodiments, the response system 188 can determine the driverstate index based on detecting a single instance of prolonged eyelidclosure or partial eyelid closure. Of course, it can also be the casethat the response system 188 analyzes eye movement over an interval oftime and looks at average eye movements.

In a further example, a response system can include provisions fordetecting the state of a driver (e.g., the behavioral state of a driver)by monitoring the head of a driver. FIG. 35 illustrates a schematic viewof a scenario in which the response system 188 is capable of monitoringthe state or behavior of a driver. Referring to FIG. 35, the ECU 106 canreceive information from an optical sensing device 162 (e.g., as part ofa head movement monitoring system 334). In some cases, the opticalsensing device 162 can be a video camera that is mounted in thedashboard of the motor vehicle 100. In other cases, a thermal sensingdevice could be used. The information can comprise a sequence of images3500 that can be analyzed to determine the state of the driver 102. Afirst image 3502 shows the driver 102 in a fully awake state, with head3504 in an upright position. However, a second image 3506 shows thedriver 102 in a drowsy state, with head 3504 leaning forward. Finally, athird image 3508 shows the driver 102 in a drowsier state with head 3504fully tilted forward. In some embodiments, the response system 188 canbe configured to analyze various images of the driver 102. Morespecifically, the response system 188 can analyze the movement of head3504 to determine if a driver is in a normal state or a drowsy (e.g.,distracted) state.

It will be understood that any type of algorithm known in the art foranalyzing head movement from images can be used. In particular, any typeof algorithm that can recognize the head and determine the position ofthe head can be used. Examples of such algorithms can include variouspattern recognition algorithms known in the art.

It is appreciated that the response system 188 can recognize other headmovements and the direction of said movements other than those describedabove. For example, as discussed above, the ECU 106 can includeprovisions for receiving information about a head pose (i.e., positionand orientation) of the driver's head. The head pose can be used todetermine what direction (e.g., forward-looking, non-forward-looking)the head of the driver is directed to with respect to the vehicle. Inone embodiment, the head movement monitoring system 334 provides headvectoring information including the magnitude (e.g., a length of time)and direction of the head look. In one embodiment, if the head pose isforward-looking, the driver is determined to be paying attention to theforward field-of-view relative to the vehicle. If the head pose isnon-forward-looking, the driver may not be paying attention.Furthermore, the head pose can be analyzed to determine a rotation ofthe head of the driver (e.g., head of driver is turned) and a rotationdirection with respect to the driver and the vehicle (i.e., to the left,right, back, forward). For example, FIG. 16B, discussed above,illustrates exemplary head looking directions of the driver with respectto the driver and the vehicle. Further, the detection of a rotation anda rotation direction can be used to recognize an eye gaze direction ofthe driver 102 as is known in the art.

It is also appreciated that the response system 188 can recognizeeye/facial movements and analyze said movements from images, similar toFIG. 35. In particular, the eye/facial movement monitoring system 332could include provisions for monitoring eye/facial movements. Eyemovement can include, for example, pupil dilation, degree of eye oreyelid closure, eyebrow movement, gaze tracking, blinking, squinting,among others. Eye movement can also include eye vectoring including themagnitude and direction of eye movement/eye gaze. Facial movements caninclude various shape and motion features of the face (e.g., nose,mouth, lips, cheeks, chin). For example, facial movements and parametersthat can be sensed, monitored and/or detected include, but are notlimited to, yawning, mouth movement, mouth shape, mouth open, the degreeof opening of the mouth, the duration of opening of the mouth, mouthclosed, the degree of closing of the mouth, the duration of closing ofthe mouth, lip movement, lip shape, the degree of roundness of the lips,the degree to which a tongue is seen, cheek movement, cheek shape, chinmovement, chin shape, etc.

FIG. 36 illustrates an embodiment of a process for detecting drowsinessby monitoring head movement in the driver. In some embodiments, some ofthe following steps could be accomplished by a response system 188 of amotor vehicle. In some cases, some of the following steps can beaccomplished by an ECU 106 of a motor vehicle. In other embodiments,some of the following steps could be accomplished by other components ofa motor vehicle, such as vehicle systems 126. In still otherembodiments, some of the following steps could be accomplished by anycombination of systems or components of the vehicle. It will beunderstood that in some embodiments one or more of the following stepscan be optional. For purposes of reference, the following methoddiscusses components shown in FIGS. 1A, 1B through 3, including theresponse system 188.

In step 3602, the response system 188 can receive optical and/or thermalinformation. In some cases, optical information could be received from acamera or an optical sensing device 162. In other cases, thermalinformation could be received from a thermal sensing device 166. Instill other cases, both optical and thermal information could bereceived from a combination of optical and thermal devices. In someembodiments, at step 3602, the response system 188 can receive headmovement information from a head movement monitoring system 334.

In step 3604, the response system 188 can analyze head movement. Bydetecting head movement, the response system 188 can determine if adriver is leaning forward. In other embodiments, the response system 188can analyze the head movement to determine a head pose of the driver'shead with respect to the driver and the vehicle as discussed above withFIGS. 16A, 16B, and 17. For example, the response system 188 candetermine a head look direction based on the head pose with respect tothe driver and the vehicle frame. As another example, the responsesystem 188 can determine a rotation (e.g., head of driver is turned)direction with respect to the driver and the vehicle (i.e., to the left,right, back, forward). Further, the response system 188 can determinehead vectoring information including a magnitude (e.g., length of time)of the head look and/or head rotation.

The head movement can be determined using either optical information,thermal information, and/or head movement information from the headmovement monitoring system 334 received during step 3602. Moreover, asdiscussed above, any type of software or algorithm can be used todetermine head movement from the optical, thermal or head movementinformation.

In step 3606, the response system 188 determines the driver state indexof the driver in response to the detected head movement. For example, insome cases, to determine the driver state index of the driver, theresponse system 188 determines if the head is tilted in any directionfor extended periods. In some cases, the response system 188 candetermine if the head is tilting forward. In some cases, the responsesystem 188 can assign a driver state index depending on the level oftilt and/or the time interval over which the head remains tilted. Forexample, if the head is tilted forward for brief periods, the driverstate index can be assigned a value of 2, to indicate that the driver isslightly drowsy (e.g., distracted). If the head is tilted forward for asignificant period of time, the driver state index can be assigned avalue of 4 to indicate that the driver is extremely drowsy (e.g.,distracted).

In some embodiments, the response system 188 can determine the driverstate index based on detecting a single instance of a driver tilting hisor her head forward. Of course, it can also be the case that theresponse system 188 analyzes head movement over an interval of time andlooks at average head movements. For example, head nods or head tiltsover a period of time.

In a further example, the response system 188 can determine the driverstate index based on detecting a head pose, head look direction and/orhead rotation. The response system 188 can also determine the driverstate index based on a length of time of the head pose and/or head lookdirection. For example, if the head look is rear-looking for more thantwo seconds, the driver state index can be assigned a value of 2, toindicate the driver is slightly drowsy (e.g., distracted). As anotherexample, if the head look is forward-looking, the driver state index canbe assigned a value of 1, to indicate the driver is not drowsy (e.g.,not distracted).

In a further example, the response system 188 can include provisions fordetecting the state of a driver by monitoring the relative position ofthe driver's head with respect to a headrest. FIG. 37 illustrates aschematic view of a scenario in which the response system 188 is capableof monitoring the state of a driver. Referring to FIG. 37, the ECU 106can receive information from a proximity sensor 184. In some cases, theproximity sensor 184 can be a capacitor. In other cases, the proximitysensor 184 can be a laser based sensor. In still other cases, any otherkind of proximity sensor known in the art could be used. The responsesystem 188 can monitor the distance between the driver's head and aheadrest 174. In particular, the response system 188 can receiveinformation from a proximity sensor 184 that can be used to determinethe distance between the driver's head and a headrest 174. For example,a first configuration 3702 shows a driver 102 in a fully awake state,with a head 186 disposed against headrest 174. However, a secondconfiguration 3704 shows the driver 102 in a somewhat drowsy state. Inthis case, the head 186 has moved further away from the headrest 174 asthe driver 102 slumps forward slightly. A third configuration 3706 showsdriver 102 in a fully drowsy state. In this case, the head 186 is movedstill further away from the headrest 174 as the driver is furtherslumped over. In some embodiments, the response system 188 can beconfigured to analyze information related to the distance between thedriver's head 186 and the headrest 174. Moreover, the response system188 can analyze head position and/or movement (including tilting,slumping, bobbing, rotation, head look) to determine if the driver 102is in a normal state or a drowsy (e.g., distracted) state.

It will be understood that any type of algorithm known in the art foranalyzing head distance and/or movement from proximity or distanceinformation can be used. In particular, any type of algorithm that candetermine the relative distance between a headrest and the driver's headcan be used. In addition, any algorithms for analyzing changes indistance to determine head motion could also be used. Examples of suchalgorithms can include various pattern recognition algorithms known inthe art.

FIG. 38 illustrates an embodiment of a process for detecting drowsinessby monitoring the distance of the driver's head from a headrest. In someembodiments, some of the following steps could be accomplished by theresponse system 188 of a motor vehicle 100. In some cases, some of thefollowing steps can be accomplished by an ECU 106 of a motor vehicle. Inother embodiments, some of the following steps could be accomplished byother components of a motor vehicle, such as vehicle systems 126. Instill other embodiments, some of the following steps could beaccomplished by any combination of systems or components of the vehicle.It will be understood that in some embodiments one or more of thefollowing steps can be optional. For purposes of reference, thefollowing method discusses components shown in FIGS. 1A, 1B through 3,including the response system 188.

In step 3802, the response system 188 can receive proximity information.In some cases, proximity information could be received from a capacitoror laser based sensor. In other cases, proximity information could bereceived from any other sensor. In step 3804, the response system 188can analyze the distance of the head from a headrest. By determining thedistance between the driver's head and the headrest, the response system188 can determine if a driver is leaning forward. Moreover, by analyzinghead distance over time, the response system 188 can also detect motionof the head. The distance of the head from the headrest can bedetermined using any type of proximity information received during step3802. Moreover, as discussed above, any type of software or algorithmcan be used to determine the distance of the head and/or head motioninformation.

In step 3806, the response system 188 determines the driver state indexof the driver in response to the detected head distance and/or headmotion. For example, in some cases, to determine the driver state indexof the driver, the response system 188 determines if the head is leaningaway from the headrest for extended periods. In some cases, the responsesystem 188 can determine if the head is tilting forward. In some cases,the response system 188 can assign a driver state index depending on thedistance of the head from the headrest as well as from the time intervalover which the head is located away from the headrest. For example, ifthe head is located away from the headrest for brief periods, the driverstate index can be assigned a value of 2, to indicate that the driver isslightly drowsy (e.g., slightly distracted). If the head is located awayfrom the headrest for a significant period of time, the driver stateindex can be assigned a value of 4 to indicate that the driver isextremely drowsy (e.g., extremely distracted). It will be understoodthat in some cases, a system could be configured so that the alert stateof the driver is associated with a predetermined distance between thehead and the headrest. This predetermined distance could be a factoryset value or a value determined by monitoring a driver over time. Then,the driver state index can be increased when the driver's head movescloser to the headrest or further from the headrest with respect to thepredetermined distance. In other words, in some cases the system canrecognize that the driver's head can tilt forward and/or backward as heor she gets drowsy.

In some embodiments, the response system 188 can determine the driverstate index based on detecting a single distance measurement between thedriver's head and a headrest. Of course, it can also be the case thatthe response system 188 analyzes the distance between the driver's headand the headrest over an interval of time and uses average distances todetermine driver state index.

In some other embodiments, the response system 188 could detect thedistance between the driver's head and any other reference locationwithin the vehicle. For example, in some cases, a proximity sensor couldbe located in a ceiling of the vehicle and the response system 188 candetect the distance of the driver's head with respect to the location ofthe proximity sensor. In other cases, a proximity sensor could belocated in any other part of the vehicle. Moreover, in otherembodiments, any other portions of a driver could be monitored fordetermining if a driver is drowsy or otherwise alert and/or distracted.For example, in still another embodiment, a proximity sensor could beused in the backrest of a seat to measure the distance between thebackrest and the back of the driver.

In another embodiment, the response system 188 could detect a positionand contact of the driver's hands on a steering wheel of the motorvehicle 100. For example, in one embodiment, the steering wheel includesa touch steering wheel system 134. Specifically, the steering wheel caninclude sensors (e.g., capacitive sensors, electrodes) mounted in or onthe steering wheel. The sensors are configured to measure contact of thehands of the driver with the steering wheel and a location of thecontact (e.g., behavioral information). In some embodiments, the sensorscan function as a switch wherein the contact of the hands of the driverand the location of the contact are associated with actuating a deviceand/or a vehicle function of the vehicle. Accordingly, the responsesystem 188 can detect and/or receive information about the positionand/or contact of the driver's hands on a steering wheel from the touchsteering wheel system 134. This information can be used to determine abehavioral driver state (e.g., a driver state index). As discussedabove, FIG. 18 illustrates an exemplary touch steering wheel 1802 withboth hands 1804 and 1806 of a driver in contact and grasping thesteering wheel.

FIG. 39 illustrates an embodiment of a process for detecting drowsinessby monitoring hand contact and position information with respect to asteering wheel. In some embodiments, some of the following steps couldbe accomplished by a response system 188 of a motor vehicle. In somecases, some of the following steps can be accomplished by an ECU 106 ofa motor vehicle. In other embodiments, some of the following steps couldbe accomplished by other components of a motor vehicle, such as vehiclesystems 126. In still other embodiments, some of the following stepscould be accomplished by any combination of systems or components of thevehicle. It will be understood that in some embodiments one or more ofthe following steps can be optional. For purposes of reference, thefollowing method discusses components shown in FIGS. 1A, 1B through 3,including the response system 188.

In step 3902, the response system 188 can receive hand contact andposition information with respect to a steering wheel. In some cases,the hand contact and position information can be received from the touchsteering wheel system 134 or directly from some kind of sensor (e.g., anoptical sensor). It is understood that in some embodiments, any type ofdriver contact information with the steering wheel can be received. Forexample, driver appendage (e.g., elbow, shoulder, arm, knee) contact andposition information. Next, in step 3904, the response system 188 cananalyze hand contact and position information. Any method of analyzinghand contact and position information can be used.

In step 3906, the response system 188 can determine the driver stateindex (e.g., a behavioral driver state index) of the driver based onhand contact and position information with respect to the steeringwheel. For example, if the driver has both hands on the steering wheel(e.g., See FIG. 18), the response system 188 can assign a driver stateindex of 1 to indicate that the driver is not distracted (e.g., notdrowsy). If the driver has one hand on the steering wheel, the responsesystem 188 can assign a driver state index of 2 to indicate that thedriver is slightly distracted (e.g., slightly drowsy). If the driver hasno hands on the steering wheel, the response system 188 can assign adriver state index of greater than 2 to indicate that the driver isdistracted (e.g., drowsy).

In some embodiments, the position of the hands can also be used todetermine the driver state index at step 3906. For example, if thedriver has both hands on the wheel, but the hands are both located at a6 o'clock steering position, the response system 188 can assign a driverstate index of 2 to indicate that the driver is slightly distracted(e.g., slightly drowsy). If the driver has both hands on the wheel,located at a 9 o'clock and 3 o'clock steering position, the responsesystem 188 can assign a driver state index of 1 to indicate that thedriver is not distracted (e.g., not drowsy). Further embodiments fordetermining a driver state and controlling a vehicle display using handcontact and position information and/or head movement information isdescribed in U.S. application Ser. No. 14/744,247 filed on Jun. 19,2015, which is incorporated herein by reference.

3. Vehicular-Sensed Driver State

A vehicular-sensed driver state is based on vehicle information fromvehicular monitoring systems and sensors, as discussed above in SectionII (B) (1). Specifically, vehicle information for determining avehicular-sensed driver state includes information related to the motorvehicle 100 of FIG. 1A and/or the vehicle systems 126, including thosevehicle systems listed in FIG. 2, that relate to a driver of the motorvehicle 100. In particular, a driver transmits information whenoperating the motor vehicle 100 and the vehicle systems 126, and basedon this operation, other types of information about the driver can beprovided by the motor vehicle 100 and/or the vehicle systems 126. Forexample, when the driver operates the motor vehicle and/or the vehiclesystems 126, changes in vehicle acceleration, velocity, lane position,and direction all provide information that directly correlates to thedriver and a state of the driver.

As an illustrative example, vehicle information for determining avehicular-sensed driver state can include steering information thatcorrelates to the driver from the electronic power steering system 132,electronic stability control system 202, the lane departure warningsystem 222, and the lane keep assist system 226, among others. Vehicleinformation for determining a vehicular-sensed driver state can includebraking information that correlates to the driver from the electronicstability control system 202, the antilock brake system 204, the brakeassist system 206, among others. Vehicle information for determining avehicular-sensed driver state can include acceleration information thatcorrelates to the driver from the electronic stability control system202, among others. Vehicle information for determining avehicular-sensed driver state can include navigation information thatcorrelates to the driver from the navigation system 230, among others.It is understood that other types of vehicle information that directlycorrelates to the driver can be obtained from other vehicle systems todetermine a vehicular-sensed driver state.

The following examples describe a variety of different methods fordetermining a vehicular-sensed driver state, for example avehicular-sensed driver state based on steering and lane departureinformation. It is understood that the methods for operating vehiclesystems in response to a vehicular-sensed driver state can also includevehicular-sensed driver states based on other types of vehicleinformation.

In one example, a response system can include provisions for detectingabnormal steering by a driver for purposes of determining if a driver isdistracted and/or drowsy. FIG. 40 illustrates a schematic view of themotor vehicle 100 being operated by a driver 102. In this situation, ECU106 can receive information related to the steering angle or steeringposition as a function of time. In addition, ECU 106 could also receiveinformation about the torque applied to a steering wheel as a functionof time. In some cases, the steering angle information or torqueinformation can be received from an EPS system 132, which can include asteering angle sensor as well as a torque sensor. By analyzing thesteering position or steering torque over time, the response system 188can determine if the steering is inconsistent, which can indicate thatthe driver is drowsy.

FIG. 41 illustrates an embodiment of a process for detecting drowsinessby monitoring the steering behavior of a driver. In some embodiments,some of the following steps could be accomplished by a response system188 of a motor vehicle. In some cases, some of the following steps canbe accomplished by an ECU 106 of a motor vehicle. In other embodiments,some of the following steps could be accomplished by other components ofa motor vehicle, such as vehicle systems 126. In still otherembodiments, some of the following steps could be accomplished by anycombination of systems or components of the vehicle. It will beunderstood that in some embodiments one or more of the following stepscan be optional. For purposes of reference, the following methoddiscusses components shown in FIGS. 1A, 1B through 3, including theresponse system 188.

In step 4102, the response system 188 can receive steering angleinformation. In some cases, the steering angle information can bereceived from EPS 132 or directly from a steering angle sensor. Next, instep 4104, the response system 188 can analyze the steering angleinformation. In particular, the response system 188 can look forpatterns in the steering angle as a function of time that suggestinconsistent steering, which could indicate a drowsy driver. Any methodof analyzing steering information to determine if the steering isinconsistent can be used. Moreover, in some embodiments, the responsesystem 188 can receive information from lane keep assist system 226 todetermine if a driver is steering the motor vehicle 100 outside of acurrent lane.

In step 4106, the response system 188 can determine the driver stateindex (e.g., a vehicular-sensed driver state index) of the driver basedon steering wheel movement. For example, if the steering wheel movementis inconsistent, the response system 188 can assign a driver state indexof 2 or greater to indicate that the driver is distracted and/or drowsy.

A response system can also include provisions for detecting abnormaldriving behavior by monitoring lane departure information. FIG. 42illustrates a schematic view of an embodiment of the motor vehicle 100being operated by a driver 102. In this situation, ECU 106 can receivelane departure information. In some cases, the lane departureinformation can be received from the LDW system 222. Lane departureinformation could include any kind of information related to theposition of a vehicle relative to one or more lanes, steering behavior,trajectory or any other kind of information. In some cases, the lanedeparture information could be processed information analyzed by the LDWsystem 222 that indicates some kind of lane departure behavior. Byanalyzing the lane departure information, the response system 188 candetermine if the driving behavior is inconsistent, which can indicatethat the driver is distracted and/or drowsy. In some embodiments,whenever the LDW system 222 issues a lane departure warning (e.g.,warning 4204), the response system 188 can determine that the driver isdrowsy. Moreover, the level of drowsiness could be determined by theintensity of the warning.

FIG. 43 illustrates an embodiment of a process for detecting drowsinessby monitoring lane departure information. In some embodiments, some ofthe following steps could be accomplished by a response system 188 of amotor vehicle. In some cases, some of the following steps can beaccomplished by an ECU 106 of a motor vehicle. In other embodiments,some of the following steps could be accomplished by other components ofa motor vehicle, such as vehicle systems 126. In still otherembodiments, some of the following steps could be accomplished by anycombination of systems or components of the vehicle. It will beunderstood that in some embodiments one or more of the following stepscan be optional. For purposes of reference, the following methoddiscusses components shown in FIGS. 1A, 1B through 3, including theresponse system 188.

In step 4302, the response system 188 can receive lane departureinformation. In some cases, the lane departure information can bereceived from the LDW system 222 or directly from some kind of sensor(such as a steering angle sensor, or a relative position sensor). Next,in step 4304, the response system 188 can analyze the lane departureinformation. Any method of analyzing lane departure information can beused.

In step 4306, the response system 188 can determine the driver stateindex (e.g., a vehicular-sensed driver state index) of the driver basedon lane departure information. For example, if the vehicle is driftingout of the current lane, the response system 188 can assign a driverstate index of 2 or greater to indicate that the driver is distractedand/or drowsy. Likewise, if the lane departure information is a lanedeparture warning from the LDW system 222, the response system 188 canassign a driver state index of 2 or greater to indicate that the driveris distracted and/or drowsy. Using this process, the response system 188can use information from one or more vehicle systems 126 to helpdetermine if a driver is drowsy. This is possible since drowsiness (orother types of inattentiveness) not only manifest as driver states, butcan also cause changes in the operation of the vehicle, which can bemonitored by the various vehicle systems 126.

It will be understood that the methods discussed above for determiningthe driver state (e.g., the driver state index) of a driver according toeye movement, head movement, steering wheel movement and/or sensingautonomic information are only intended to be exemplary and in otherembodiments any other method of detecting the state of a driver,including states associated with drowsiness, could be used. For example,driver state can be determined by monitoring heart rate informationand/or information transfer rates as discussed herein.

Additionally, it will be understood that the method discussed above fordetermining driver states can also be used for determining a pluralityof driver states and/or a combined driver state. Specifically, it willbe understood that in some embodiments multiple methods for detectingdriver states to determine a driver state could be used simultaneously,as will now be discussed in detail.

B. Determine Combined Driver State

As discussed above, FIG. 24A illustrates an embodiment of a process forcontrolling one or more vehicle systems in a motor vehicle based on thestate of the driver. However, in one embodiment, controlling one or morevehicle systems in a motor vehicle can depend on one or more driverstates (e.g., a plurality of driver states), specifically, a combineddriver state based on one or more driver states. The “combined driverstate,” as used herein, refers to a combined measure of the state of thedriver, for example the vigilance, the attention and/or the drowsinessof a driver. In some cases, the combined driver state could be given asa numerical value, for example a combined driver state level, a combineddriver state index, among others. In other cases, the combined driverstate could be given as a non-numerical value, for example, drowsy,non-drowsy, slightly drowsy, a Boolean value, among others. Moreover,the combined driver state can range from values associated with completealertness (e.g., attentive) to values associated with extreme drowsiness(e.g., distraction) or even a state in which the driver is asleep (e.g.,distraction). For example, in one embodiment, the combined driver stateindex could take on the values 1, 2, 3 and 4, where 1 is the leastdrowsy and 4 is the most drowsy. In another embodiment, the combineddriver state index could take on values from 1-10. In other cases, thecombined driver state can range from values associated with completefocus on the driving task (10 for example) to values associated completedistraction (1 for example) and values there between.

The one or more driver states can be one of a physiological driverstate, a behavioral driver state and a vehicular-sensed driver state.Thus, the combined driver state can be based on different types ofdriver states derived from different types of monitoring information(e.g., physiological information, behavioral information, vehicleinformation) and/or from information from different types of monitoringsystems (e.g., physiological monitoring systems and sensors, behavioralmonitoring systems and sensors, vehicular monitoring systems andsensors). The combined driver state can also be based on the same typesof driver states or various combinations of driver states that can bederived from the same or different types of monitoring informationand/or monitoring systems.

Further, the one or more driver states can be determined, combinedand/or and confirmed with one another. Determining, combining and/orconfirming one or more driver states provides a reliable and robustdriver monitoring system. This driver monitoring system verifies driverstates (e.g., to eliminate false positives), provides a combined driverstate based on more than one driver state using different types ofmonitoring information (e.g., multi-modal inputs), and modifies one ormore vehicle systems based on the combined driver state. In this way,behaviors and risks can be assessed in multiple modes and modificationof vehicle systems can be controlled accurately.

1. Determine Combined Driver State Based on a Plurality Driver States

Referring now to FIG. 44, a method is illustrated of an embodiment of aprocess for controlling one or more vehicle systems in a motor vehicle,similar to FIG. 24A, except the process of FIG. 44 depends on a combineddriver state based on a plurality of driver states. In some embodiments,some of the following steps could be accomplished by a response system188 of a motor vehicle. In some cases, some of the following steps canbe accomplished by an ECU 106 of a motor vehicle. In other embodiments,some of the following steps could be accomplished by other components ofa motor vehicle, such as vehicle systems 126. In still otherembodiments, some of the following steps could be accomplished by anycombination of systems or components of the vehicle. It will beunderstood that in some embodiments one or more of the following stepscan be optional. For purposes of reference, the following methoddiscusses components shown in FIGS. 1A, 1B through 3, including theresponse system 188.

In step 4402, the response system 188 can receive monitoringinformation. In one embodiment, the monitoring information is at leastone of physiological information, behavioral information and vehicleinformation. The monitoring information can be received from one or moresensors, one or more monitoring systems, one or more vehicle systems,any other device of the motor vehicle 100, and/or any combination ofsensors, monitoring systems, vehicles systems or other devices.

In step 4404, the response system 188 can determine a plurality ofdriver states. The plurality of driver states being at least one of aphysiological driver state, a behavioral driver state, or avehicular-sensed driver state. A physiological driver state, abehavioral driver state, or a vehicular-sensed driver state can bereferred to herein as drive state types or types of driver states. Thephysiological driver state is based on physiological information, thebehavioral driver state is based on behavioral information, and thevehicular-sensed driver state is based on vehicle information. As willbe discussed herein, in some embodiments, each of the plurality ofdriver states are a different one of a physiological driver state, abehavioral driver state, or a vehicular-sensed driver state. In otherembodiments, at least two of the plurality of driver states are based onthe same type of driver state.

In some embodiments, step 4404 includes determining a first driver stateand a second driver state based on the monitoring information from theone or more monitoring systems. In another embodiment, step 4404includes determining a third driver state based on the monitoringinformation from the one or more monitoring systems. It is appreciatedthat other combinations of information and driver states can beimplemented. For example, behavioral information could be used todetermine a first driver state and physiological information could beused to determine a second driver state, and so on. In another example,first and second driver states could be based on behavioral informationfrom two different systems or sensors.

It is appreciated that any number of driver states can be determined. Inone embodiment, the first driver state is one of a physiological driverstate, a behavioral driver state, or a vehicular-sensed driver state,and the second driver state is another of a physiological driver state,a behavioral driver state, or a vehicular-sensed driver state. The thirddriver state can be a further one of a physiological driver state, abehavioral driver state, or a vehicular-sensed driver state. By usingdifferent types of monitoring information to determine different driverstates, multi-modal driver state confirmation is possible, as will bedescribed herein. The driver state can be determined by the ECU 106, theresponse system 188, the vehicle systems 126 and/or the monitoringsystems 300 described herein.

It should be noted that in any of the embodiments described herein, thefirst, second and third driver states could each be derived from thesame type of monitoring systems and/or information, meaning the firstdriver state could be a physiological driver state based onphysiological information, the second driver state could be aphysiological driver state based on physiological information butderived from a different source than the first driver state, and thethird driver state could be a physiological driver state based onphysiological information but derived from a different source thaneither the first or second driver state. In addition, the first, secondand third driver states could each be derived from distinct monitoringsystems and/or information, meaning the first driver state could be aphysiological driver state based on physiological information, thesecond driver state could be a behavioral driver state based onbehavioral information and the third driver state could be avehicular-sensed driver state based on vehicle information. Anycombination of these examples is possible.

In step 4406, the response system 188 can determine a combined driverstate based on the plurality of driver states of step 4404. In somecases, the combined driver state can be normal or drowsy. In othercases, the combined driver state can range over three or more statesranging between normal and very drowsy (or even asleep). As will bediscussed in further detail herein, the combined driver state can bedetermined in various ways.

In step 4408, in some embodiments, the response system 188 can determinewhether the driver state is true based on the combined driver state. Forexample, whether or not the driver is vigilant, drowsy, inattentive,distracted, intoxicated, among others. If the driver state is not true(i.e., NO), the response system 188 can proceed back to step 4402 toreceive additional monitoring information. If, however, the driver stateis true (i.e., YES), the response system 188 can proceed to step 4410.

In step 4410, the response system 188 can modify the control of one ormore vehicle systems, including any of the vehicle systems discussedabove. By modifying the control of one or more vehicle systems, theresponse system 188 can help to avoid various hazardous situations thatcan be caused by, for example, a distracted and/or drowsy driver. Insome embodiments, step 4408 is optional and after determining a combineddriver state at step 4406, the method can directly proceed to step 4410,where modifying the control of the one or more vehicle systems is basedon the combined driver state. FIG. 25, discussed above, illustratesvarious vehicle systems and how these vehicle systems can be modified orcontrolled by the response system 188.

As discussed above, FIG. 26 illustrates an embodiment of a process ofmodifying the operation of a vehicle system according to the level ofdrowsiness detected. However, in one embodiment, modifying the operationof a vehicle system can depend on a plurality of driver state levels. Inparticular, the plurality of driver state levels can be combined into acombined driver state level. Each of the plurality of driver statelevels can be one of a physiological driver state level, a behavioraldriver state level and a vehicular-sensed driver state level. Thus, thecombined driver state level can be based on different types of driverstate levels each derived from different types of monitoring informationand/or from information from different types of monitoring systems.

Referring now to FIG. 45, a method is illustrated of an embodiment of aprocess for controlling one or more vehicle systems in a motor vehiclethat depends on a combined driver state level based on a plurality ofdriver state levels. In step 4502, the response system 188 can determinea plurality of driver state levels. In one embodiment, each of theplurality of driver state levels is based on at least one ofphysiological information, behavioral information, and vehicleinformation. Thus, the plurality of driver state levels are at least oneof a physiological driver state level, a behavioral driver state levelor a vehicular-sensed driver state level. Said differently, thephysiological driver state level is based on physiological information,the behavioral driver state level is based on behavioral information andthe vehicular-sensed driver state level is based on vehicle information.

The driver state level can be a “level of drowsiness.” The term “levelof drowsiness” as used throughout this detailed description and in theclaims refers to any numerical or other kind of value for distinguishingbetween two or more states of drowsiness. For example, in some cases,the level of drowsiness can be given as a percentage between 0% and100%, where 0% refers to a driver that is totally alert and 100% refersto a driver that is fully drowsy or even asleep. In other cases, thelevel of drowsiness could be a value in the range between 1 and 10. Instill other cases, the level of drowsiness is not a numerical value, butcould be associated with a given discrete state, such as “not drowsy,”“slightly drowsy,” “drowsy,” “very drowsy” and “extremely drowsy.”Moreover, the level of drowsiness could be a discrete value or acontinuous value.

In another embodiment, the driver state level can be a “level ofdistraction.” The term “level of distraction” as used throughout thisdetailed description and in the claims refers to any numerical or otherkind of value for distinguishing between two or more states ofdistraction. For example, in some cases, the level of distraction can begiven as a percentage between 0% and 100%, where 0% refers to a driverthat is totally attentive and 100% refers to a driver that is fullydistracted. In other cases, the level of distraction could be a value inthe range between 1 and 10. In still other cases, the level ofdistraction is not a numerical value, but could be associated with agiven discrete state, such as “not distracted,” “slightly distracted,”“distracted”, “very distracted” and “extremely distracted”. Moreover,the level of distraction could be a discrete value or a continuousvalue. In some cases, the level of distraction can indicate the driveris engaged in a secondary task (e.g., other than the primary task ofdriving).

In some cases, the level of drowsiness and/or distraction can beassociated with a driver state index. Thus, in some embodiments, in step4504, the response system 188 can determine a plurality of driver stateindices. In one embodiment, each of the driver state indices are basedon at least one of physiological information, behavioral information,and vehicle information. The term “driver state index” refers to ameasure of the state of driver, for example, the level drowsiness of adriver and/or the level distraction of the driver. In some cases, thedriver state index could be given as a numerical value. In other cases,the driver state index could be given as a non-numerical value.Moreover, the driver state index can range from values associated withcomplete alertness (e.g., attentive) to values associated with extremedrowsiness (e.g., extreme distraction) or even a state in which thedriver is asleep. In one embodiment, the driver state index could takeon the values 1, 2, 3 and 4, where 1 is the least drowsy (e.g.,distracted) and 4 is the most drowsy (e.g., distracted). In anotherembodiment, the driver state index could take on values from 1-10.

Accordingly, at step 4504, the plurality of driver state levels areleast one of a physiological driver state, a behavioral driver state ora vehicular-sensed driver state. Said differently, the physiologicaldriver state level is based on physiological information, the behavioraldriver state level is based on behavioral information and thevehicular-sensed driver state level is based on vehicle information.

In some embodiments, step 4504 includes determining a first driver statelevel and a second driver state level based on the monitoringinformation from the one or more monitoring systems. In anotherembodiment, the step 4504 includes determining a third driver statelevel based on the monitoring information from the one or moremonitoring systems. It is appreciated that other combinations ofinformation and driver state levels can be implemented. For example,behavioral information could be used to determine a first driver statelevel and physiological information could be used to determine a seconddriver state level, and so on.

It is appreciated that any number of driver state levels can bedetermined. In one embodiment, the first driver state level is one of aphysiological driver state level, a behavioral driver state level or avehicular-sensed driver state level, and the second driver state levelis another of a physiological driver state level, a behavioral driverstate level or a vehicular-sensed driver state level. The third driverstate level can be a further one of a physiological driver state level,a behavioral driver state level or a vehicular-sensed driver statelevel. By using different types of monitoring information to determinedifferent driver states, multi-modal driver state confirmation ispossible, as will be described herein. The driver state levels can bedetermined by the response system 188, the vehicle

In step 4506, the response system 188 can determine a combined driverstate level based on the plurality of driver state levels of step 4504.In another embodiment, in step 4506, the response system 188 candetermine a combined driver state index based on the plurality of driverstate indices of step 4504. As will be discussed in further detailherein, the combined driver state can be determined in various ways.

In step 4508, in some embodiments, the response system 188 can determinewhether or not the driver state is true based on the combined driverstate level and/or index. For example, whether or not the driver isvigilant, drowsy, inattentive, distracted, intoxicated, among others. Ifthe driver state is not true (i.e., NO), the response system 188 canproceed back to step 4502 to receive additional monitoring information.If, however, the driver state is true (i.e., YES), the response system188 can proceed to step 4510.

In step 4510, the response system 188 can modify the control of one ormore vehicle systems, including any of the vehicle systems discussedabove. By modifying the control of one or more vehicle systems, theresponse system 188 can help to avoid various hazardous situations thatcan be caused by, for example, a drowsy and/or distracted driver. Insome embodiments, step 4508 is optional and after determining a combineddriver state at step 4506, the method can directly proceed to step 4510,where modifying the control of the one or more vehicle systems is basedon the combined driver state.

In another embodiment and with reference to FIG. 46, driver states canbe determined and combined into one or more groups. In step 4602, theresponse system 188 can determine a plurality of driver states, and insome embodiments, driver state levels. At step 4604, the method includesdetermining a first combined driver state based on the plurality ofdriver states of step 4602. In this embodiment, the first combineddriver state can be based on a subset of the plurality of driver states.For example, at step 4604, a first driver state, a second driver state,a third driver state and a fourth driver state can be determined.Accordingly, at step 4606, the first combined driver state can be basedon a subset of the plurality of driver states, for example, the firstdriver state and the second driver state. In other embodiments, thefirst combined driver state is based on the first driver state and thethird driver state, or any other combination.

At step 4608, the method can include determining a second combineddriver state. The second combined driver state can be based on the firstcombined driver state and one or more other driver states. For example,if the first combined driver state is based on the first driver stateand the second driver state, the second combined driver state can bebased on the first combined driver state, the third driver state and thefourth driver state. It is appreciated, that other combinations ofdriver states and combined driver states can be implemented. Further, itis appreciated that a second set of a plurality of driver states can bedetermined at step 4608. In this embodiment, the second combined driverstate can be based on the first combined driver state and the second setof plurality of driver states.

In step 4508, in some embodiments, the response system 188 can determinewhether or not the driver state is true based on the combined driverstate level and/or index. For example, whether or not the driver isvigilant, drowsy, inattentive, distracted, intoxicated, among others. Ifthe driver state is not true (i.e., NO), the response system 188 canproceed back to step 4602 to receive additional monitoring information.If, however, the driver state is true (i.e., YES), the response system188 can proceed to step 4612.

At step 4612, the vehicle systems can be controlled based on the firstcombined driver state and/or the second combined driver state. It isappreciated, that although FIG. 46 illustrates two combined driverstates, the process can include more than two combined driver states.

As discussed above with FIG. 27, in some embodiments, the responsesystem 188 can determine a control parameter. In one embodiment, thecontrol parameter can be based on the combined driver state leveldetermined by the response system 188 in step 4506 of FIG. 45. The term“control parameter” as used throughout this detailed description and inthe claims refers to a parameter used by one or more vehicle systems. Insome cases, a control parameter can be an operating parameter that isused to determine if a particular function should be activated for agiven vehicle system. The control parameter can be used in step 4506 tomodify the control of one or more vehicle systems.

Determining a control parameter based on the combined driver state leveland/or index will now be discussed. FIG. 47 illustrates a schematic viewof how a combined driver state index can be used to retrieve a controlcoefficient. A control coefficient can be any value used in determininga control parameter. In some cases, the control coefficient varies as afunction of driver state index and is used as an input for calculatingthe control parameter. Examples of control coefficients include, but arenot limited to electronic stability control system coefficients, brakeassist coefficients, blind spot zone warning coefficients, warningintensity coefficients, forward collision warning coefficients, lanedeparture warning coefficients and lane keep assist coefficients. Somesystems cannot use a control coefficient to determine the controlparameter. For example, in some cases, the control parameter can bedetermined directly from the driver state index.

In one embodiment, the value of the control coefficient 4702 increasesfrom 0% to 25% as the combined driver state index increases from 1 to 4.In some cases, the control coefficient can serve as a multiplicativefactor for increasing or decreasing the value of a control parameter.For example, in some cases when the combined driver state index is 4,the control coefficient can be used to increase the value of a controlparameter by 25%. In other embodiments, the control coefficient couldvary in any other manner. In some cases, the control coefficient couldvary linearly as a function of the combined driver state index. In othercases, the control coefficient could vary in a nonlinear manner as afunction of the combined driver state index. In still other cases, thecontrol coefficient could vary between two or more discrete values as afunction of the combined driver state index.

FIG. 29, discussed above, illustrates a calculation unit 2902 fordetermining a control parameter. The calculation unit 2902 receives acontrol coefficient 2904 and vehicle operating information 2906 asinputs. The calculation unit 2902 outputs the control parameter 2908.The vehicle operating information 2906 can include any informationnecessary to calculate a control parameter. For example, in situationswhere the vehicle system is an electronic stability control system, thesystem can receive wheel speed information, steering angle information,roadway friction information, as well as other information necessary tocalculate a control parameter that is used to determine when stabilitycontrol should be activated. Moreover, the control coefficient 2904 canbe determined from the combined driver state index using, for example, alook-up table. The calculation unit 2902 then considers both the vehicleoperating information and the control coefficient 2904 in calculatingthe control parameter 2908.

In some embodiments, a control parameter can be associated with a statusor state of a given vehicle system. FIG. 48 illustrates an embodiment ofa general relationship between the combined driver state index of thedriver and a system status 4802. The system shown here is general andcould be associated with any vehicle system. For a low combined driverstate index (1 or 2), the system status 4802 is ON. However, if thecombined driver state index increases to 3 or 4 the system status 4802is turned OFF. In still other embodiments, a control parameter could beset to multiple different “states” according to the combined driverstate index. Using this arrangement, the state of a vehicle system canbe modified according the combined driver state index of a driver.

i. Exemplary Driver State Combinations

Determining a combined driver state and/or a combined driver state indexwill now be described in further detail. It is appreciated that thefollowing combinations can be implemented with the systems and methodsfor confirming driver states as will be discussed below. FIG. 49illustrates an exemplary AND logic gate 4902 that can be executed by theresponse system 188 for combining a plurality of driver states, namely,a first driver state (DS₁) and a second driver state (DS₂). It isunderstood that any number of driver states can be combined (e.g.,DS_(i) . . . DS_(n)). Further, as discussed above, it is understood thata driver state can also be a driver state index. In FIG. 49, each driverstate is determined based on one of a plurality of monitoringinformation types, namely, physiological information, behavioralinformation, and vehicle information. Accordingly, the first driverstate and the second driver state are each one a physiological driverstate, a behavioral driver state, or a vehicular-sensed driver state. Inparticular, in one embodiment, the first driver state, and the seconddriver state are each a different one of said driver states. In anotherembodiment, the first driver state and the second driver state can bethe same type (i.e., behavioral) but derived from different monitoringsystems and/or information.

At the AND logic gate 4902, the response system 188 analyzes the firstdriver state and the second driver state to determine a combined driverstate. In the illustrative examples discussed herein, drowsiness will beused as an exemplary driver state, however, it is understood that otherdriver states can be implemented. For example, if the first driver state(e.g., a physiological driver state) indicates a drowsy driver state(i.e., YES; 1) and the second driver state (e.g., a vehicular-senseddriver state) indicates a drowsy driver state (i.e., YES; 1), thecombined driver state returned by the gate 4902 indicates a drowsydriver state (i.e., YES; 1), based on the first driver state and thesecond driver state. In another example, if the first driver state(e.g., a behavioral driver state) indicates a non-drowsy driver state(i.e., NO; 0), and the second driver state (e.g., a physiological driverstate) indicates a drowsy driver state (i.e., YES; 1), the combineddriver state returned by the gate 4902 indicates a non-drowsy driverstate (i.e., NO; 0), based on the first driver state and the seconddriver state.

A truth table 4904 illustrates the various combinations and functionsfor the AND logic gate 4902. Although the AND logic gate 4902 isdescribed with Boolean values, it is understood that in otherembodiments, which will be described herein, the first driver state, thesecond driver state and the combined driver state can each includenumeric values (e.g., a driver state index, a combined driver stateindex). Thus, the response system 188 can determine a combined driverstate based on the first driver state numeric value and/or the seconddriver state numeric value as a result of the output of the AND logicgate 11700.

FIG. 50 illustrates another exemplary AND logic gate 5002 for combininga plurality of driver states. In this example, a first driver state(DS₁), a second driver state (DS₂) and a third driver state (DS₃) arecombined. Similar to FIG. 49, each of the driver states is determinedbased on one of a plurality of monitoring information types, namely,physiological information, behavioral information, and vehicleinformation. Accordingly, the first driver state, the second driverstate and the third driver state, are one a physiological driver state,a behavioral driver state and a vehicular-sensed driver state. However,it is understood that in other embodiments, the one or more of thedriver states can be based on physiological information, behavioralinformation, and vehicle information.

At the AND logic gate 5002, the response system 188 analyzes the firstdriver state, the second driver and the third driver state inputs todetermine a combined driver state. For example, if the first driverstate (e.g., a physiological driver state) indicates a drowsy driverstate (i.e., YES; 1), the second driver state (e.g., a vehicular-senseddriver state) indicates a drowsy driver state (i.e., YES; 1), and thethird driver state (e.g., a behavioral driver state) indicates a drowsydriver state (i.e., YES; 1), the combined driver state returned by thegate 5002 indicates a drowsy driver state (i.e., YES; 1), based on thefirst driver state, the second driver state and the third driver state.In another example, if the first driver state (e.g., a behavioral driverstate) indicates a non-drowsy driver state (i.e., NO; 0), the seconddriver state (e.g., a physiological driver state) indicates a drowsydriver state (i.e., YES; 1), and the third driver state (e.g., avehicular-sensed driver state) indicates a drowsy driver state (i.e.,YES; 1), the combined driver state returned by the gate 5002 indicates anon-drowsy driver state (i.e., NO; 0), based on the first driver state,the second driver state and the third driver state. A truth table 5004illustrates the various combinations and functions for the AND logicgate 5002.

Although the AND logic gate 5002 is described with Boolean values, it isunderstood that in other embodiments, which will be described herein,the first driver state, the second driver state, the third driver stateand the combined driver state can each include numeric values (e.g., adriver state index, a combined driver state index). Thus, the responsesystem 188 can determine a combined driver state based on the firstdriver state numeric value, the second driver state numeric value and/orthe third driver state numeric value as a result of the output the ANDlogic gate 5002.

FIG. 51 illustrates an exemplary AND/OR logic gate 5102 that can beexecuted by the response system 188 for combining a plurality of driverstates, namely, a first driver state (DS₁), a second driver state (DS₂)and a third driver state (DS₃). Similar to FIGS. 49 and 50, each of thedriver states is determined based on one of a plurality of monitoringinformation types, namely, physiological information, behavioralinformation, and vehicle information. Accordingly, the first driverstate, the second driver state and the third driver state, are one aphysiological driver state, a behavioral driver state and avehicular-sensed driver state. However, it is understood that in otherembodiments, the one or more of the driver states can be based onphysiological information, behavioral information, and vehicleinformation.

At the AND/OR logic gate 5102, the response system 188 analyzes thefirst driver state, the second driver and the third driver state inputsto determine a combined driver state. The AND/OR logic gate 5102,includes an OR logic gate 5104 to analyze a first driver state and asecond driver state and an AND logic gate 5106 to analyze an output ofthe OR logic gate 5104 and the third driver state. For example, if thefirst driver state (e.g., a physiological driver state) indicates adrowsy driver state (i.e., YES; 1), and the second driver state (e.g., avehicular-sensed driver state) indicates a drowsy driver state (i.e.,YES; 1), the output of the OR logic gate 5104 indicates a drowsy driverstate (i.e., YES; 1). Accordingly, if the third driver state (e.g., abehavioral driver state) indicates a drowsy driver state (e.g., YES; 1),the combined driver state returned by the gate 5106 indicates a drowsydriver state (e.g., YES; 1), based on the first driver state, the seconddriver state and the third driver state.

In another example, if the first driver state (e.g., a vehicular-senseddriver state) does not indicate a drowsy driver state (i.e., NO; 0), andthe second driver state (e.g., a physiological driver state) indicates adrowsy driver state (i.e., YES; 1), the output of the OR logic gate 5104indicates a non-drowsy driver state (i.e., NO; 0). Accordingly, if thethird driver state (e.g., a behavioral driver state) indicates a drowsydriver state (i.e., YES; 1), the combined driver state returned by thegate 5106 indicates a drowsy driver state (i.e., YES; 1), based on thefirst driver state, the second driver state and the third driver state.In some embodiments, the combined driver state can be based on onlythose driver states that indicate a drowsy driver state (i.e., YES; 1).Thus, in the previous example, the combined driver state can be based onthe second driver state and the third driver state.

A truth table 5108 illustrates the various combinations and functions ofthe AND/OR logic gate 5102. Although the AND/OR logic gate 5102 isdescribed with Boolean values, it is understood that in otherembodiments, which will be described herein, the first driver state, thesecond driver state, the third driver state, and the combined driverstate can each include numeric values (e.g., a driver state index, acombined driver state index). Thus, the response system 188 candetermine a combined driver state based on the first driver statenumeric value, the second driver state numeric value and/or the thirddriver state numeric value as a result of the output the AND/OR logicgate 5102.

ii. Exemplary Combined Driver State Calculations

As mentioned above, each driver state (e.g., a physiological driverstate, a behavioral driver state, and a vehicular-sensed driver state)and the combined driver state can be quantified as a level, a numericvalue or a numeric value associated with a level. For example, as adriver state level, a combined driver state level, a driver state index,a combined driver state index, among others. Based on the methods,examples and logic gates described above in FIGS. 44-51, the combineddriver state can be computed in various ways. In the examples thatfollow, each driver state will be quantified as a driver state index andthe combined driver state will be quantified as a combined driver stateindex, however, it is appreciated that other combinations orquantifications are contemplated.

In one embodiment, the response system 188 determines a combined driverstate index by aggregating each driver state index (i.e., each driverstate). For example, the combined driver state index I is the sum of oneor more driver state indices as follows:I=Σ _(i=1) ^(n) DS _(i)  (10)Where I is the combined driver state index and DS_(i) is the driverstate index for DS_(i) . . . DS_(n). In one embodiment, each driverstate index DS_(i) is one of a plurality of driver states (e.g., aphysiological driver state, a behavioral driver state, avehicular-sensed driver state). As an illustrative example, withreference to the AND logic gate 5002 of FIG. 50, let DS₁=5 (i.e., aphysiological driver state index) indicating a drowsy driver state(i.e., YES; 1), DS₂=6 (i.e., a behavioral driver state index) indicatinga drowsy driver state (i.e., YES; 1), and DS₃=4 (i.e., avehicular-sensed driver state index) indicating a drowsy driver state(i.e., YES; 1). The AND logic gate 5002 returns a combined driver stateindex indicating a drowsy driver state (i.e., YES; 1). Accordingly, theresponse system 188 computes the combined driver state index usingequation (1) as 15 (5+6+4).

In some embodiments, the combined driver state could be based onselecting driver states that return a YES value (i.e., indicating adrowsy driver state). As an illustrative example, with reference to theAND/OR logic gate 5102 of FIG. 51, let DS₁=2 (i.e., a physiologicaldriver state index) indicating a non-drowsy driver state (i.e., NO; 0),DS₂=6 (i.e., a behavioral driver state index) indicating a drowsy driverstate (i.e., YES; 1), and DS₃=4 (i.e., a vehicular-sensed driver stateindex) indicating a drowsy driver state (i.e., YES; 1) The AND/OR logicgate 5102 returns a combined driver state indicating a drowsy driverstate (i.e., YES; 1). Accordingly, the response system 188 computes thecombined driver state index using equation (1) and based on DS₂ and DS₃,as 10 (6+4). It is understood, that in other embodiments, the combineddriver state index can be based on each driver state index, regardlessof whether the driver state index indicates a drowsy driver state.

In another embodiment, the response system 188 determines a combineddriver state index as an average of each driver state index. Forexample, the combined driver state index I is the average of one or moredriver state indices as follows:

$\begin{matrix}{I = \frac{\sum\limits_{i = 1}^{n}\;{DS}_{i}}{n}} & (11)\end{matrix}$Where k is the combined driver state index and DS_(i) is the driverstate index for DS_(i), . . . DS_(n). Similar to the illustrativeexamples describing equation (10), the combined driver state accordingto equation (11) can be based on each driver state or based on eachdriver state that returns a YES value (i.e., indicating a drowsy driverstate).

In a further embodiment, the response system 188 determines a combineddriver state index as a weighted average of each driver state index. Forexample, the combined driver state index I is the weighted average ofone or more driver state indices as follows:

$\begin{matrix}{I = \frac{\sum\limits_{i = 1}^{n}\;{{DS}_{i}w_{i}}}{\Sigma_{i}^{n}w_{i}}} & (12)\end{matrix}$Where I is the combined driver state index and DS_(i) is the driverstate index for DS_(i) . . . DS_(n). The weight of each driver stateindex can be based on different factors. In one embodiment, the weightof each driver state index is based on the type of driver state, thetype of monitoring information and/or the type of monitoring system andsensors. In another embodiment, the weight of each driver state index isbased on the quality of the monitoring information (e.g., signalstrength). In a further embodiment, the weight of each driver stateindex is based on a location or a placement of the monitoring systemsand sensors. In some embodiments, the weight of each driver state indexcan be pre-determined and/or based on the identity of the driver. Inother embodiments, the weight of each driver state index can bedynamically selected or learned using artificial intelligence. In otherembodiments, the weight of each driver state index is based on aconfidence score of the applicable system or the data received from theapplicable system.

It is understood that various selections of driver states andcombinations of driver states can be implemented with the methodsdiscussed above. In some embodiments, the selections of driver statesand combinations of driver states can be determined using artificialintelligence, such as a neural network. Further, it is understood thatthe exemplary combinations and computations described above can be usedin whole or in part with the methods discussed below.

2. Determine Combined Driver State with Threshold Comparisons

In one embodiment, determining the combined driver state includescomparing at least one of the plurality of driver states to a threshold.Specifically, in some cases, determining the combined driver statefurther includes comparing at least one of the plurality of driverstates to a threshold, and upon determining the at least one of theplurality of driver states meets the threshold, determining the combineddriver state based on the least one of the plurality of driver states.Thus, for example, upon determining that a first driver state meets afirst driver state threshold and a second driver state meets a seconddriver state threshold, the combined driver state is determined based onthe first driver state and the second driver state.

The term “threshold” as used throughout this detailed description and inthe claims refers to any numerical or other kind of value used forcomparison with another value to determine one or more driver states,confirm one or more driver states, combine one or more driver states,modify one or more vehicles systems, determine or modify a controlparameter, a control coefficient, or a failsafe threshold, among others.In some cases, the threshold is given as a percentage, a value between 1and 10, a discrete value, a continuous value, or a range of values. Thethreshold can also be a frequency or a function of time. As will bediscussed in more detail herein, the thresholds can be pre-determinedand dynamically modified based on the driver states, the monitoringinformation, and/or the identity of the driver.

FIG. 52 illustrates a method of an embodiment of a process forcontrolling one or more vehicle systems in a motor vehicle, similar toFIG. 45, except the process of FIG. 52 includes threshold comparisons.The method of FIG. 52 includes at step 5202 receiving monitoringinformation. At step 5204, the method includes determining a pluralityof driver state levels (e.g., DS_(i) . . . DS_(n)) based on themonitoring information. In one embodiment, each driver state isassociated with a threshold related to said driver state. For example, afirst driver state DS_(i) can be associated with a first driver statethreshold T_(i). Accordingly, in FIG. 52, at step 5206, for each driverstate (e.g., while i>0), it is determined if the driver state DS_(i)meets the threshold T_(i). If so (i.e., YES), DS_(i) is stored, forexample, in an array at step 5208, and a counter X is incremented. Onceeach driver state is compared to its associated threshold, at step 5210,it is determined if X is greater than 0. If so (i.e., YES), the storeddriver states which met the associated thresholds are used to determinea combined driver state at step 5212. If not (i.e., NO, none of thedriver states met the associated threshold), the method can return tostep 5202 to receive monitoring information.

Referring now to FIG. 53, the exemplary AND logic gate of FIG. 50 isshown as AND logic gate 5302 with threshold logic (i.e., T₁, T₂, T₃).The threshold can be related to the driver state and/or the monitoringinformation used to determine the driver state. As an illustrativeexample, if the first driver state DS₁ is based on heart rate (i.e.,physiological information), the first driver state threshold T_(i) canbe a numeric value indicating a high heart rate. It is appreciated thatthe thresholds described above can be applied to any number of driverstates, to any of the logic gates discussed above and the confirmationof one or more driver states discussed below.

As mentioned above, the thresholds can be pre-determined and dynamicallymodified based on the driver states, the information used to determinethe driver state (e.g., heart information, head pose information), thetype of information and/or driver state (e.g., physiological,behavioral, vehicular), other types of monitoring information, and/orthe identity of the driver. Accordingly, the thresholds provide anaccurate measurement for the specific driver state, driver, and drivingenvironment for determining the driver states, combining the driverstates, and confirming the driver states. Illustrative embodiments willnow be discussed.

In one embodiment, the thresholds can be determined and/or dynamicallychanged based on the monitoring information received, for example, fromthe systems shown in FIGS. 2 and 3. As mentioned above, the thresholdcan be related to the driver state and/or the monitoring informationused to determine the driver state. As an illustrative example, if thefirst driver state is based on contiguous heart rate accelerations ordecelerations, the first driver state threshold may be a numeric valueindicating a high number of contiguous heart rate accelerations ordecelerations. For example, a high number of contiguous heart rateaccelerations or decelerations can be associated with a numeric value of13.

Accordingly, the threshold can be related to a pattern of monitoringinformation. For example, the driver state can be number indicating apattern and/or frequency of monitoring information over a period oftime. Thus, the threshold can be a value associated with the patternover a period of time. In one embodiment, the driver state can be basedon steering information, for example, steering information indicatingjerks, and/or steering corrections over a period of time. Accordingly,the threshold can be set to a value to determine whether the pattern ofsteering jerks over a period of time indicates a drowsy or non-drowsydriver. As an illustrative example, a threshold of 10 jerks in 30seconds can indicate a drowsy driver.

In another embodiment, the driver state can be a number indicating lanedepartures over a period of time. Accordingly, the threshold can be setto a value to determine whether the number of lane departures over aperiod of time indicate a drowsy or non-drowsy driver. In anotherembodiment, the driver state can be a number indicating the number ofacceleration and decelerations over a period of time. Accordingly, thethreshold can be set to a value to determine whether the numberacceleration and decelerations over a period of time indicate a drowsyor non-drowsy driver.

In another embodiment, the driver state can be a number indicating afrequency of head nods (e.g., the number of head nods over a period oftime). Accordingly, the threshold can be set to a value to determinewhether the frequency of head nods over a period of time indicate adrowsy or non-drowsy driver. In another example, the driver state can bea number of head looks from a forward-looking direction to anon-forward-looking direction (e.g., looking at a navigation system).Accordingly, the threshold can be set to a value to determine whetherthe driver is attentive or distracted. For example, a threshold of 10head looks can indicate an inattentive driver.

In another embodiment, the threshold can indicate a pattern and/orfrequency of monitoring information over time including vectoring (e.g.,magnitude/length of time, direction) information about the head, eyesand/or body of the driver. For example, a driver state can be a numberof head looks from a forward-looking direction to a head lookingdirection directed to a navigation system, where the head lookingdirection has a magnitude (e.g., time length) of a pre-determined numberof seconds. Accordingly, the threshold can be set to a value todetermine whether the driver is attentive or distracted based on thehead vectoring, for example, five head looks.

As discussed above, the thresholds can be dynamically modified based onmonitoring information. For example, a threshold can be dynamicallymodified based on gesture information from the gesture recognition andmonitoring system 330. For example, if it is determined based on thegesture information that the driver is operating a portable device intheir hand, a threshold can be automatically adjusted to account forthis risk. Thus, a threshold indicating an inattentive driver statecould be lowered. As another example, if the driver's breathing isdetermined to be irregular based on information from the respiratorymonitoring system 312, a threshold indicating a stressed driver statecould be lowered.

In another embodiment, the threshold can be modified based on contactand position information of the driver's hands with the steering wheel.For example, the threshold can be modified based on information from thetouch steering wheel system 134. In another example, a threshold relatedto a driver state based on perspiration rate information, can beadjusted based on monitoring information from the vehicle systems 126.For example, the monitoring information from a climate control systemmay indicate the internal temperature of the vehicle is hot. If theinternal temperature of the vehicle is hot, the driver can naturallyhave a higher perspiration rate. Thus, perspiration rate may not be anaccurate indication of a driver state and the associated threshold maybe increased.

Additionally, as mentioned above, the thresholds can be pre-determinedand/or modified based on the identity of the driver and characteristicsof the identified driver. For example, the response system 188 candetermine the identity of the driver based on monitoring information,for example, from the systems of FIG. 3 as discussed in Section III (B)(4). In some embodiments, systems and methods of biometricidentification (FIGS. 22-23) can be used to identify the driver andstore normative data and/or past and current thresholds associated withthe driver. It is appreciated that the response system 188 can use amachine pattern learning method to track monitoring information for theidentified driver and determine normative baseline data for theidentified driver. Any machine learning method or pattern recognitionalgorithm could be used. The normative baseline data can be used todetermine the thresholds and/or modified the thresholds for theidentified driver. Further, average, and/or normative data for otherdrivers with similar characteristics of the identified driver (e.g.,age, sex) can be used to determine the thresholds and/or modified thethresholds for the identified driver. Accordingly, the thresholds areadaptive and learned overtime and/or are controlled based on theidentity of the driver.

In one embodiment, the driver state can be a number indicating thenumber of acceleration and decelerations over a period of time.Accordingly, the response system 188, after identifying the driver, canmodify the threshold related to the number of acceleration anddecelerations over a period of time based on the particular drivinghabits of the driver. For example, the driver's baseline data may showthat the driver typically has a high number of accelerations anddecelerations. Accordingly, the threshold can be modified to account forthe driver's baseline data. For example, the threshold for indicating adrowsy driver may be increased.

Referring again to the illustrative example above where a threshold is anumeric value indicating a high number of contiguous heart rateaccelerations or decelerations, the baseline threshold can be set to anumeric value of 13 to indicate a drowsy driver. However, after trackingthe data of the identified driver, the numeric value of 13 may notindicate a drowsy driver state for the identified driver. Accordingly,the system may modify the value to 15.

In another embodiment, the response system 188 can determine that thenormative baseline heart rate of a particular driver is higher than anaverage adult heart rate. Accordingly, the response system 188 candynamically modify the threshold related to heart rate for the driverbased on the driver normative baseline heart rate. In anotherembodiment, the response system 188 can determine an age of the driverbased on the identity of the driver. For example, the response system188, after determining the identity of the driver, can retrieve a userprofile including characteristics user preferences for the identifieddriver. The characteristics can include the age of the driver. Theresponse system 188 can modify and/or determine the threshold based onthe age of the driver. For example, a threshold associated with analcohol level may be decreased (e.g., providing more strict control bylowering the alcohol level needed to reach the threshold) for a youngdriver.

In another embodiment, the response system 188 can determine that one ormore vehicle occupants are present in the vehicle. The response system188 can modify the threshold levels for the driver based on determiningthat one or more vehicle occupants are present. For example, a vehiclespeed threshold may be lowered since to provide more safety for theother vehicle occupants present in the vehicle. In another embodiment,the response system 188 can identify the one or more vehicle occupantspresent in the vehicle and modify the threshold based on acharacteristic of the one or more vehicle occupants. For example, if oneof the vehicle occupants is young (e.g., a baby), the thresholds can bemodified As discussed above, in one embodiment, the driver state can bebased on steering information, for example, steering informationindicating jerks and/or steering corrections over a period of time.Accordingly, after identifying a young vehicle occupant is present inthe vehicle, the response system 188 can modify the threshold (e.g.,decrease) for determine whether the pattern of steering jerks over aperiod of time indicates a drowsy driver.

Referring now to FIG. 54, a general process for determining and/ormodifying a threshold is shown. At step 5402, the method includesreceiving monitoring information. In some embodiments, at step 5402, themethod can also include receiving and/or determining a driver state(e.g., based on the monitoring information). At step 5404, the methodincludes identifying the driver, for example, using the methods andsystems discussed in Section III (B) (4). Step 5404 can also include, atstep 5408, receiving stored driver data. The stored driver data caninclude monitoring information tracked over time (e.g., using machineand pattern learning algorithms). The stored driver data can be receivedusing the telematics control unit from the Internet, a network, astorage device located at a network, among others.

At step 5406, the method includes modifying and/or determining athreshold based on the identity of the driver. More specifically, theresponse system 188 can analyze the stored driver data to determinepatterns of the identified driver and modify and/or determine thethreshold accordingly. It is understood that the exemplary driverstates, thresholds, and modifications discussed above are exemplary andother driver states, thresholds, and modifications can be implemented.

In some embodiments, the process shown in FIG. 54 can apply todetermining and/or modifying a control parameter and/or a controlcoefficient. Thus, at step 5406 of FIG. 54, the method can includemodifying a control parameter and/or a control coefficient of one ormore vehicle systems based on the identified driver. As an illustrativeexample, in situations where a lane deviation warning system is used,the control parameter can be a distance threshold to a potential lanedeviation to provide a warning to the driver. Based on the identity ofthe driver and tracking the data of the identified driver (e.g., thestored driver data), the response system 188 may determine that theidentified driver tends to drive close to the lane markers. Accordingly,the response system 188 can modify the control parameter based on theidentified driver. For example, the response system 188 can decrease thedistance threshold to a potential lane deviation to account for theidentified driver's tendency to driver close to the lane markers.

As another illustrative example, in situations where an electronicstability control system is used, the control coefficient can be astability error of steering associated with under-steering orover-steering. Based on the identity of the driver and tracking the dataof the identified driver (e.g., the stored driver data), the responsesystem 188 may determine that the identified driver naturally driverwith a slight over-steer. Accordingly, the response system 188 canmodify stability error of steering associated over-steering based on theidentified driver. For example, the response system 188 can decrease thestability error of steering associated over-steering to account for theidentified driver's slight over-steer. In some embodiments, the controlcoefficient can be modified as function of the pattern associated withthe identified driver. For example, if the driver naturally drives witha moderate over-steer, the response system 188 can decrease thestability error of steering associated over-steering more than, if thedriver naturally drives with a slight over-steer. Similarly, the controlparameter can be modified as a function of the pattern associated withthe identified driver.

3. Determine Combined Driver State with Confirmation of One or MoreDriver States

In one embodiment, the system and methods for responding to driver stateinclude confirming one or more driver states with other driver states todetermine a combined driver state. Said differently, the response system188 can confirm at least one selected of the plurality of driver stateswith at least one selected different one of the plurality of driverstates and determine a combined driver state index based on the at leastone selected of the plurality of driver states and the at least oneselected different one of the plurality of driver states.

The term “confirming,” as used herein can include comparing two valuesto validate the state of the driver. Accordingly, a first driver statecan be confirmed with a second driver state by comparing the firstdriver state to the second driver state and determining if the firstdriver state and the second driver state both indicate the same orsubstantially the same driver state.

FIG. 55 illustrates a method of an embodiment of a process forcontrolling one or more vehicle systems in a motor vehicle withconfirming one or more driver states to determine a combined driverstate. At step 5502, the method includes receiving monitoringinformation. At step 5504, the method includes determining a pluralityof driver states. In one example, determining a plurality of driverstates can include determining a first driver state and a second driverstate. In some cases, each state of the plurality of driver states isdetermined based on one of a plurality of monitoring information types,namely, physiological information, behavioral information, and vehicleinformation. Accordingly, in one example, the first driver state and thesecond driver state are one of a physiological driver state, abehavioral driver state, and a vehicular-sensed driver state.

At step 5506, the method includes confirming at least one selected ofthe plurality of driver states with at least one selected different oneof the plurality of driver states. In one embodiment, confirmingincludes comparing the at least one selected of the plurality of driverstates with the at least one selected different one of the plurality ofdriver states. For example, in FIG. 55, the first driver state DS₁ isconfirmed with the second driver state DS₂. If the first driver state isa physiological driver state indicating a drowsy driver (i.e., YES; 1)and the second driver state is a behavioral driver state indicating adrowsy driver (i.e., YES; 1), then at step 5508, the combined driverstate would indicate a drowsy driver. In another example, if the firstdriver state is a physiological driver state indicating a drowsy driver(i.e., YES; 1) and the second driver state is a vehicular-sensed driverstate indicating a non-drowsy driver (i.e., NO; 1), then at step 5508,the combined driver state would indicate a non-drowsy driver. In someembodiments if the output of the confirmed driver state is NO, then theprocess may proceed back to step 5502 to receive monitoring information.It is understood that steps 5506 and 5508 could be processed using thelogic gates of FIGS. 49, 50, and 51. It is also understood that themethod of FIG. 55 can apply to a state or a level of state. For example,determining a driver state, a driver state level, a combined driverstate, and a combined driver state level.

In further embodiment, the response system 188 can confirm at least onedriver state of the plurality of driver states with another one of theplurality of driver states and combine the at least one driver state ofthe plurality of driver states with the another one of the plurality ofdriver states. As discussed above, and referring again to FIG. 55, atstep 5506, the method includes confirming at least one selected of theplurality of driver states with at least one selected different one ofthe plurality of driver states. In one embodiment, confirming includescomparing the at least one selected of the plurality of driver stateswith the at least one selected different one of the plurality of driverstates. For example, if the first driver state is a physiological driverstate indicating a drowsy driver (i.e., YES; 1) and the second driverstate is a behavioral driver state indicating a drowsy driver (i.e.,YES; 1), then at step 11306, determining a combined driver state caninclude determining the combined driver state based on the first driverstate and the second driver state. For example, determining the combineddriver state can include aggregating the first driver state and thesecond driver state, calculating an average of the first driver stateand the second driver state, calculating a weighted average of the firstdriver state and the second driver state, and so on. It is understoodthat steps 5506 and 5508 could be processed using the logic gates ofFIGS. 49, 50, and 51. It is also understood that the method of FIG. 55can apply to a state or a level of state. For example, determining adriver state, a driver state level, a combined driver state, and acombined driver state level.

In some embodiments confirming one driver state with one or more driverstates to determine a combined driver state can include comparing saiddriver states to a particular threshold as discussed above withreference to FIG. 52. FIG. 56 illustrates a method of an embodiment of aprocess for controlling one or more vehicle systems in a motor vehiclewith confirming one or more driver states to determine a combined driverstate including thresholds.

At step 5602, the method includes receiving monitoring information. Atstep 5604, the method includes determining a plurality of driver states.At step 5606, the method includes confirming at least one selected ofthe plurality of driver states with at least one selected different oneof the plurality of driver states. In one embodiment, confirmingincludes comparing the at least one selected of the plurality of driverstates with the at least one selected different one of the plurality ofdriver states.

At step 5608, for each confirmed driver state (e.g., while i>0), it isdetermined if the confirmed driver state DS_(i) meets the thresholdT_(i). If so (i.e., YES), DS_(i) is stored, for example, in an array atstep 5610, and a counter X is incremented. Once each confirmed driverstate is compared to its associated threshold, at step 5612, it isdetermined if X is greater than 0. If so (i.e., YES), the stored driverstates which met the associated thresholds are used to determine acombined driver state at step 5614. If not (i.e., NO; none of the driverstates met the associated threshold hold), the method can return to step5602 to receive monitoring information. As an illustrative example, inFIG. 56, the first driver state DS₁ is confirmed with the second driverstate DS₂. If the first driver state is a physiological driver stateindicating a drowsy driver (i.e., YES; 1) and the second driver state isa behavioral driver state indicating a drowsy driver (i.e., YES; 1),then at step 5614, the combined driver state would indicate a drowsydriver. It is understood that steps 5606 and 5614 could be processedusing the logic gates of FIGS. 49, 50, and 51. It is also understoodthat the method of FIG. 56 can apply to a state or a level of state. Forexample, determining a driver state, a driver state level, a combineddriver state, and a combined driver state level.

FIG. 57 illustrates another embodiment of a method of a process forcontrolling one or more vehicle systems in a motor vehicle withconfirming one or more driver states to determine a combined driverstate including thresholds. In the embodiment of FIG. 57, at step 5702the method includes receiving monitoring information. At step 5704, themethod includes determining a plurality of driver states. At step 5706,for each driver state (e.g., while i>0), it is determined if the driverstate DS_(i) meets the threshold T_(L). If so (i.e., YES), DS_(i) isstored, for example, in an array at step 5708, and a counter X isincremented. If not, (i.e., NO), the method can end and return to step5704.

Once each driver state is compared to its associated threshold, at step5710, it is determined if X is greater than 0. If not (i.e., NO; none ofthe driver states met the associated threshold hold), the method canreturn to step 5702 to receive monitoring information. If so (i.e.,YES), one or more of the stored driver states that met the associatedthresholds are confirmed at step 5712. Specifically, the method includesconfirming at least one selected of the plurality of driver states withat least one selected different one of the plurality of driver states.In one embodiment, confirming includes comparing the at least oneselected of the plurality of driver states with the at least oneselected different one of the plurality of driver states. In FIG. 57,the first driver state DS₁ is confirmed with the second driver stateDS₂. For example, if the first driver state is a physiological driverstate indicating a drowsy driver (i.e., YES; 1) and the second driverstate is a behavioral driver state indicating a drowsy driver (i.e.,YES; 1), then at step 5714, determining a combined driver state caninclude determining the combined driver state based on the first driverstate and the second driver state. For example, determining the combineddriver state can include aggregating the first driver state and thesecond driver state, calculating an average of the first driver stateand the second driver state, calculating a weighted average of the firstdriver state and the second driver state, and so on. It is understoodthat steps 5712 and 5714 could be processed using the logic gates ofFIGS. 49, 50, and 51. It is also understood that the method of FIG. 57can apply to a state or a level of state. For example, determining adriver state, a driver state level, a combined driver state, and acombined driver state level.

FIG. 58 illustrates another embodiment of a method of a process forcontrolling one or more vehicle systems in a motor vehicle withconfirming one or more driver states to determine a combined driverstate including thresholds. In the embodiment of FIG. 58, at step 5802the method includes receiving monitoring information. At step 5804, themethod includes determining a plurality of driver states. At step 5806,for each driver state (e.g., while i>0), it is determined if the driverstate DS_(i) meets the threshold T_(L). If so (i.e., YES), DS_(i) isstored, for example, in an array at step 5808, and a counter X isincremented. If not (i.e., NO), the process can return to step 5804).

Once each driver state is compared to its associated threshold, at step5810, it is determined if X is greater than 0. If not (i.e., NO; none ofthe driver states met the associated threshold hold), the method canreturn to step 5802 to receive monitoring information. If so (i.e.,YES), one or more of the stored driver states that met the associatedthresholds are confirmed at step 5812.

Specifically, at step 5812 the method includes confirming at least oneselected of the plurality of driver states with at least one selecteddifferent one of the plurality of driver states. In one embodiment,confirming includes comparing the at least one selected of the pluralityof driver states with the at least one selected different one of theplurality of driver states. In FIG. 58, the first driver state DS₁ isconfirmed with the second driver state DS₂. In another embodiment, theoutcome of the confirmation of the first driver state DS₁ and the seconddriver state DS₂. is confirmed with the third driver state DS₃. Forexample, if the first driver state is a physiological driver stateindicating a drowsy driver (i.e., YES; 1) and the second driver state isa behavioral driver state indicating a drowsy driver (i.e., YES; 1),then the outcome the confirmation of the first driver state and thesecond driver state indicates a drowsy driver state (i.e., YES; 1). Theoutcome can be compared to the third driver state. If the third driverstate is a vehicular-sensed driver state and indicates a drowsy driver(i.e., YES; 1), then at step 5814, the combined driver state canindicate a drowsy driver. However, if the third driver state is avehicular-sensed driver state and indicates a non-drowsy driver (i.e.,NO; 0), then at step 5814, the combined driver state can indicate anon-drowsy driver.

In another embodiment, determining the combined driver state can includeaggregating, calculating an average, or calculating a weighted averageof the first driver state, the second driver state, and the third driverstate. It is understood that steps 5812 and 5814 could be processedusing the logic gates of FIGS. 49, 50, and 51. It is also understoodthat the method of FIG. 58 can apply to a state or a level of state. Forexample, determining a driver state, a driver state level, a combineddriver state, and a combined driver state level. It should also beunderstood that any of the embodiments described above for determining acombined driver state can apply to a state, a level of state, or a stateindex. In other words, a combined driver state index could be foundusing the methods described above.

In some embodiments, the driver state confirmation processes describedabove can include assigning a priority level to the driver states andconfirming the driver states in an order based on the priority level.The priority level can be based on the type of driver state, the driverstate level, the type of monitoring information the driver state isbased on, the quality of the monitoring information, among others. Inthis way, the driver state confirmation process can be controlled andprovide accurate confirmation results. Referring now to FIG. 59, anembodiment of a method of a process for controlling one or more vehiclesystems in a motor vehicle with confirming one or more driver statesbased on priority levels to determine a combined driver state is shown.

In the embodiment of FIG. 59, at step 5902 the method includes receivingmonitoring information. At step 5904, the method includes determining aplurality of driver states. At step 5906, the method includes assigninga priority level to each driver state determined at step 5904. Thepriority level can be based on the type of driver state, the driverstate level, the type of monitoring information the driver state isbased on, the quality of the monitoring information, among others. Thepriority level indicates an order for confirming the driver states. Asan illustrative example, a first driver state DS₁ can be assigned apriority level of 4, a second driver state DS₂ can be assigned apriority level of 1, a third driver state DS₃ can be assigned a prioritylevel of 2 and a fourth driver state DS₄ can be assigned a prioritylevel of 3. In this example, the driver states can be confirmed with oneanother, at step 5908, in order of the priority level, for example, thesecond driver state DS₂, the third driver state DS₃, the first driverstate DS₁ and the fourth driver state DS₄, where a priority level of 1is the highest priority level.

As another illustrative example, the priority level can be based on thetype of monitoring information used to determine each driver state. Forexample, in one embodiment, assigning a priority level to each driverstate at step 5906 is based on the type of monitoring information, inthe following order from highest to lowest priority level: physiologicalmonitoring information, behavioral monitoring information and vehicularmonitoring information. Further, priority levels can be assigned basedon the characteristic used to determine the monitoring information. Forexample, physiological information can be assigned a priority level inthe following order from highest to lowest: heart monitoringinformation, eye movement information, and head movement information. Inboth of these examples, the priority level is based on the type ofinformation and the type of characteristic wherein an internalcharacteristic receives a higher priority level and an externalcharacteristic.

In another embodiment, the priority level can be based on the quality ofthe monitoring information used to determine each driver state. Forexample, the signals indicating a measurement of the mentoringinformation can be analyzed to determine the quality of the signals.Monitoring information with a high quality signal (e.g., no/less noise)can be assigned a higher priority level than monitoring information witha low quality signal (e.g., high noise). The methods for selectivelyreceiving output from sensors and processing the output as described inin U.S. Pat. No. 9,398,875, entitled A System and Method for BiologicalSignal Analysis, filed on Nov. 7, 2013, which is incorporated byreference in its entirety herein, discussed above, can be used to assignpriority levels to monitoring information based on the quality of themonitoring information.

Similarly, in some embodiments, at step 5906, the method can includeselectively confirming driver states based on the priority level. Forexample, a driver state with a low priority level can be discarded andnot used in the confirmation process. As an illustrative example, driverstates based on monitoring information having a high quality signal(e.g., no/less noise) can be assigned a higher priority level thandriver states based on monitoring information with a low quality signal(e.g., high noise). In this example, driver states with a low prioritylevel (e.g., indicating low quality monitoring information) areselectively discarded and not used during the confirmation process.

It is understood that steps 5908 and 5910 could be processed using thelogic gates of FIGS. 49, 50, and 51. It is also understood that themethod of FIG. 59 can apply to a state or a level of state. For example,determining a driver state, a driver state level, a combined driverstate, and a combined driver state level. It should also be understoodthat any of the embodiments described above for determining a combineddriver state can apply to a state, a level of state, or a state index.In other words, a combined driver state index could be found using themethods described above.

4. Network System for Determining a Combined Driver State

The components of the systems and methods described above for combiningand confirming one or more driver states can be organized into differentarchitectures for different embodiments. Referring now to FIG. 60, adiagram of network system 6000 for controlling one or more vehiclesystems including confirming and combining one or more driver statesaccording to an exemplary embodiment is shown. The system 6000 can insome embodiments, be an artificial neural network for controlling one ormore vehicle systems. Additionally, it is understood that the systemsand methods described above for combining and confirming one or moredriver states can be implemented with the system 6000.

The vehicle systems 126 (FIG. 1A, 2) and/or the monitoring systems 300(FIG. 3) discussed above provide monitoring information to the system6000. The monitoring information can include physiological information6002, behavioral information 6004, and/or vehicle information 6006. Byutilizing physiological information 6002, behavioral information 6004and vehicle information 6006, the network system 6000 is created thatdetermines more than one type of driver state to accurately assess thedriver and a current vehicle situation and subsequently control one ormore vehicle systems appropriately. As shown in FIG. 60, physiologicalinformation 6002, behavioral information 6004 and vehicle information6006 can be used to determine an input node, namely, determine a firstdriver state 6008, determine a second driver state 6010 and determine athird driver state 6012. It is appreciated, and as will be discussed infurther detail herein, other numbers of driver states, for example, two,three, four, five, etc., can be used.

In one exemplary embodiment, the first driver state 6008 is based onphysiological information 6002, for example, heart rate measured by aheart rate sensor (e.g., a bio-monitoring sensor 180) positioned in thevehicle seat 168 of the motor vehicle 100 (FIG. 1A). The second driverstate 6010 can be based on behavioral information 6004, for example,pupil dilation measured by an optical sensor, for example, the opticalsensing device 162 in the motor vehicle 100 (FIG. 1). The third driverstate 6012 can be based on vehicle information 6006, for example,steering information from the electronic power steering system 132(FIGS. 1 and 2). The above types of exemplary driver states areillustrative in nature and it is appreciated that other types ofphysiological information 6002, behavioral information 6004, and vehicleinformation 6006 can be used to determine one or more of the driverstates.

In the embodiment of FIG. 60, as shown, each input node (e.g., driverstate) can include a threshold related to said node. For example, thefirst driver state 6008 can have a first driver state threshold that isrelated to the first driver state. Thus, for example, if the firstdriver state 6008 is based on a heart rate numeric value, the firstdriver state threshold may be a numeric value indicating a high heartrate.

As discussed above, the thresholds can be pre-determined for each driverstate and/or the information the driver state is based on. In otherembodiments, the threshold is also determined based on the particulardriver and adjusted based on the driver. For example, the responsesystem 188 can use a machine learning method to determine normativebaseline data for a particular driver. Any machine learning method orpattern recognition algorithm could be used. For example, the responsesystem 188 can determine that the normative baseline heart rate of aparticular driver is higher than an average adult heart rate.Accordingly, the response system 188 can dynamically modify thethreshold related to heart rate for the driver based on the drivernormative baseline heart rate. As discussed above, the threshold can becustomized based on the driver. In some embodiments, systems and methodsof biometric identification (FIGS. 22-23) can be used to identify thedriver and store normative data and/or past and current thresholdsassociated with the driver.

As discussed above, the threshold can be dynamically modified orpre-determined based on other monitoring information. As an illustratedexample, a threshold related to a driver state based on perspirationrate information, can be adjusted based on monitoring information fromthe vehicle systems 126 indicating the internal temperature of thevehicle is hot. If the internal temperature of the vehicle is hot, thedriver can naturally have a higher perspiration rate, which may not bean accurate indication of a driver state. Accordingly, the thresholdrelated to a driver state based on perspiration rate information can bedynamically modified to account of the internal temperature of thevehicle. Other examples of thresholds and modifying thresholds arediscussed in Section III (B) (2).

In one embodiment, upon determining one or more driver states at theinput nodes, the input nodes are activated thereby triggering the outputnodes to determine a combined driver state index based on the one ormore driver states. For example, the first driver state 6008, the seconddriver state 6010 and/or the third driver state 6012 are combined into acombine driver state index. In some cases, the one or more driver statesare first compared to the associated threshold before determining acombine driver state index. Upon meeting said threshold, the one or moredriver states are combined into a combined driver state index at theoutput nodes.

In another embodiment, upon meeting said threshold, the input node isactivated and subsequently triggers activation at a confirmation node.This allows at least one selected of the plurality of driver states tobe confirmed with at least one selected different one of the pluralitydriver states. Thus, for example, confirmation node 6014 triggersconfirmation of the first driver state 6008 with the second driver state6010 and/or the third driver state 6012. More specifically, in oneembodiment, upon meeting the first driver state threshold, the seconddriver state 6010 is compared to the second driver state threshold. Uponmeeting the second driver state threshold, in one embodiment the thirddriver state 6012 is compared to the third driver state threshold. Inother embodiments, upon meeting the first driver state threshold, thethird driver state 6012 is compared to the second driver statethreshold, and so forth. It is appreciated that other combinations ofconfirmation can be implemented.

Similarly, confirmation node 6016 triggers confirmation of the seconddriver state 6010 with the first driver state 6008 and/or the thirddriver state 6012. Confirmation node 6018 triggers confirmation of thethird driver state 6012 with the first driver state 6008 and/or thesecond driver state 6010. Accordingly, by confirming more than onedriver state based on more than one type of monitoring information,accurate driver state estimation is possible.

Moreover, the confirmed driver states can be forwarded to an outputnode. Specifically, a combined driver state index is determined based onthe confirmed driver states and the combined driver state index isoutput to control one or more vehicle systems. As discussed above, thecombined driver state index can be determined in various ways. Forexample, by aggregation, averaging, or weighted averaging.

As an illustrative example, at confirmation node 6014, the first driverstate 6008 was confirmed with the second driver state 6010 and the thirddriver state 6012. Accordingly, the combined driver state index atoutput node 6020 can be determined as an aggregate of the first driverstate 6008, the second driver state 6010 and the third driver state6012. If for example, the first driver state 6008 was confirmed with thesecond driver state 6010, the combined driver state index at output node6020 can be determined as an aggregate of the first driver state 6008and the second driver state 6010.

Referring now to FIG. 61 a schematic flow chart of a detailed process ofcontrolling vehicle systems according to a combined drive state indexaccording to the network 6000 of FIG. 60 is shown. As shown in FIG. 61,monitoring information is received and includes receiving physiologicalinformation at step 6102, receiving behavioral information at step 6104,and receiving vehicle information at step 6106. Specifically, in oneembodiment, the monitoring information is at least one of physiologicalinformation, behavioral information or vehicle information. Thephysiological information received at step 6102 is input used todetermine a first driver state at step 6108. The behavioral informationreceived at step 6104 is input used to determine a second driver stateat step 6110. The vehicle system information received at step 6106 isused to determine a third driver state at step 6112. It is appreciatedthat other combinations of information and driver states can beimplemented. For example, behavioral information could be used todetermine a first driver state and physiological information could beused to determine a second driver state, and so on. It is appreciatedthat other combinations of information and driver states can beimplemented. For example, behavioral information could be used todetermine a first driver state and physiological information could beused to determine a second driver state, and so on.

It is appreciated that any number of driver states can be determined. Inone embodiment, the first driver state is one of a physiological driverstate, a behavioral driver state, or a vehicular-sensed driver state,and the second driver state is another of a physiological driver state,a behavioral driver state, or a vehicular-sensed driver state. The thirddriver state can be a further one of a physiological driver state, abehavioral driver state, or a vehicular-sensed driver state. By usingdifferent types of monitoring information to determine different driverstates, multi-modal driver state confirmation is possible, as will bedescribed herein. Further, it is appreciated that the plurality ofdriver states can be determined in various ways as discussed throughoutthe specification. For example, the driver state could be a driver stateindex. The driver state can be determined by the response system 188,vehicle systems 126 and/or the monitoring systems described herein.

As discussed above, in some embodiments, determining the combined driverstate index further includes comparing at least one of the plurality ofdriver states to a threshold, and upon determining the at least one ofthe plurality of driver states meets the threshold, determining thecombined driver state index based on the least one of the plurality ofdriver states. Thus, for example, upon determining that a first driverstate meets a first driver state threshold and a second driver statemeets a second driver state threshold, the combined driver state indexis determined based on the first driver state and the second driverstate. In another embodiment, as discussed above, determining thecombined driver state index further includes confirming at least oneselected of the plurality of driver states with at least one selecteddifferent one of the plurality of driver states and determining acombined driver state index based on the at least one selected of theplurality of driver states and the at least one selected different oneof the plurality of driver states.

As illustrated in the example shown in FIG. 61, at step 6114, it isdetermined if the first driver state meets the first driver statethreshold. Upon determining that the first driver state meets the firstdriver threshold, first driver state is confirmed with at least oneother driver state at step 6122. For example, in one embodiment, thefirst driver state is confirmed with the second driver state, forexample, at step 6116. In this embodiment, the first driver state is oneof a physiological driver state, a behavioral driver state, or avehicular-sensed driver state, and the second driver state is another ofa physiological driver state, a behavioral driver state, or avehicular-sensed driver state. Accordingly, the state of the driver isassessed by confirming driver states based on different monitoringinformation types.

As mentioned above, confirming the first driver state with the seconddriver state can further include comparing the second driver state to asecond driver state threshold at step 6116. Upon determining the seconddriver state meets the second driver state threshold, the method caninclude determining the combined driver state index based on the firstdriver state and the second driver state at step 6128.

In another embodiment, the step of confirming includes confirming thefirst driver state with at least one of the second driver state or thethird driver state and determining the combined driver state index basedon the first driver state and the at least one of the second driverstate or the third driver state. For example, upon determining that thefirst driver state meets the first driver state threshold at step 6114,the first driver state is confirmed at step 6122 with the third driverstate at step 6118. Upon determining that the third driver state meetsthe third driver state threshold, a combined driver state index isdetermined at step 6128 based on the first driver state and the thirddriver state.

It will be appreciated that in some embodiments, all three driver statesare confirmed (e.g., by determining if each driver state meets itsrespective driver state threshold) and the combined driver state indexis based on all three driver states. In this example, the three driverstates are each one of a physiological driver state, a behavioral driverstate, or a vehicular-sensed driver state. Further, as discussed above,the thresholds discussed in FIG. 61 can be pre-determined and/ordynamically based on the driver states, the information used todetermine the driver state (e.g., heart information, head poseinformation), the type of information and/or driver state (e.g.,physiological, behavioral, vehicular), other types of monitoringinformation, and/or the identity of the driver.

Determining which driver state triggers the confirmation process andwhich driver states are confirmed in response can be based on anartificial neural network, for example, the network 6000 of FIG. 60. Forexample, determining which driver state triggers the confirmationprocess and which driver states are confirmed can be predetermined basedon the type of monitoring information, type of driver distraction and/ordynamically selected.

For example, in one embodiment, to determine if a driver is drowsy, thedriver states could be based on predetermined monitoring informationindicating a drowsy driver, for example, heart rate from a heart ratesensor placed on the driver's seat to determine a first driver state,eye movement information from an optical sensor to determine a seconddriver state, and steering information from the steering wheel todetermine a third driver state. In other embodiments, the driver statescould be dynamically selected based on the quality of the monitoringinformation. For example, if it is determined that the heart rateinformation is weak (e.g., using signal analysis), a driver state basedon a different type of physiological information could be determined.

V. Determine One or More Vehicular States

In addition to determining one or more driver states, in someembodiments, the systems and methods for responding to driver state canalso include determining one or more vehicular states and modifying thecontrol of one or more vehicle systems based on the driver state and/orthe vehicular state, or any combination of one or more of said states. Avehicular state describes a state of the motor vehicle 100 and/or thevehicle systems 126. In particular, in some embodiments, the vehicularstate describes a state of the motor vehicle 100 based on externalinformation about the vehicle environment. In one embodiment, thevehicular state can describe a risk surrounding the vehicle environment.For example, as discussed below in Section B, a vehicular state can becharacterized as a hazard, a hazard level, a risk level, among others.

A vehicular state is based on vehicle information from vehicularmonitoring systems and sensors, as discussed above in Section III (B)(1). Specifically, vehicle information for determining a vehicular stateincludes information related to the motor vehicle 100 of FIG. 1A and/orthe vehicle systems 126, including those vehicle systems listed in FIG.2. As an illustrative example, vehicle information for determining avehicular state can include information about objects, pedestrians,hazards, and/or other vehicles in the environment of the vehicle, forexample from visual devices 140, the collision warning system 218, theautomatic cruise control system 216, the lane departure warning system222, the blind spot indicator system 224, the lane keep assist system226, the lane monitoring system 228, among others. Vehicle informationfor determining a vehicular state can include traffic information,weather information, road speed limit information, navigationinformation, for example, from, visual devices 140, the climate controlsystem 234, and the navigation system 230, among others.

In another embodiment, the vehicular state can be based on a failuredetection system. For example, the failure detection system 244 candetect a level of failure and/or a fail-safe state of the motor vehicle100 and/or vehicle systems 126. Vehicle information for determining avehicular state can also include other information corresponding to themotor vehicle 100 and/or the vehicle systems 126 describing the state ofthe motor vehicle 100 and/or the external environment of the motorvehicle 100.

Similar to the driver state discussed above, it is understood that thevehicular state can also be quantified as a level, a numeric value or anumeric value associated with a level. In some embodiments, discussedabove, the vehicular state can be characterized as a hazard, a type ofhazard, a hazard level, and/or a risk level. In one embodiment,controlling one or more vehicle systems is based on one or more driverstates and one or more vehicular states. Referring now to FIG. 62, amethod is illustrated of an embodiment of a process for controlling oneor more vehicle systems in a motor vehicle similar to FIG. 45, however,the process is based on a combined driver state level and a vehicularstate.

At step 6202, the method includes receiving monitoring information. Instep 6204, the response system 188 can determine a plurality of driverstate levels. In one embodiment, each of the plurality of driver statelevels is based on at least one of physiological information, behavioralinformation, and vehicle information. Thus, the plurality of driverstate levels are at least one of a physiological driver state level, abehavioral driver state level or a vehicular-sensed driver state level.Said differently, the physiological driver state level is based onphysiological information, the behavioral driver state level is based onbehavioral information, and the vehicular-sensed driver state level isbased on vehicle information.

In step 6206, the response system 188 can determine a combined driverstate level based on the plurality of driver state levels of step 6204.In another embodiment, in step 6206, the response system 188 candetermine a combined driver state index based on the plurality of driverstate indices of step 6204. As will be discussed above, the combineddriver state can be determined in various ways.

In step 6208, in some embodiments, the response system 188 can determinewhether or not the driver state is true based on the combined driverstate level and/or index. For example, whether or not the driver isvigilant, drowsy, inattentive, distracted, intoxicated, among others. Ifthe driver state is not true (i.e., NO), the response system 188 canproceed back to step 6202 to receive additional monitoring information.If, however, the driver state is true (i.e., YES), the response system188 can proceed to step 6210.

At step 6210, the response system 188 can determine a vehicular state.As discussed above, the vehicular state can be based on vehicleinformation. In another embodiment, the response system 188 candetermine more than one vehicular state. In one embodiment, the processproceeds to step 6212. In another embodiment, the response system 188proceeds to step 6214, where the response system 188 compares the driverstate level to the vehicular state level. In another embodiment, insteadof comparing the driver state level to the vehicular state level, theresponse system 188 compares the vehicular state level to apredetermined threshold. The predetermined threshold can be based on thevehicular state and/or the vehicle information used to determine thevehicular state. If the outcome of step 6214 is YES, the response systemcan proceed to step 6212. If the outcome of step 6214 is NO, theresponse system 188 can proceed back to step 6202 to receive additionalmonitoring information.

In step 6212, the response system 188 can automatically modify thecontrol of one or more vehicle systems, including any of the vehiclesystems discussed above, based on the driver state level and thevehicular state. By automatically modifying the control of one or morevehicle systems, the response system 188 can help to avoid varioushazardous situations that can be caused by, for example, a drowsydriver.

It is understood that the vehicular state and/or a vehicular state levelcan be determined before or after other steps shown in FIG. 62. Forexample, in some embodiments, the vehicular state can be determined atstep 6204. Further, in other embodiments, the vehicular state can beused to determine a combined driver state level, for example, as shownin FIGS. 49, 50 and 51. It is appreciated that the logic gates,equations and methods described in Section IV can also be implementedwith a fourth state, the vehicular state.

VI. Modify Control of Vehicle Systems

As discussed above, in some embodiments, modifying control of one ormore vehicle systems can be based on a driver state, a level of a driverstate, a driver state index, a combined driver state, a level of acombined driver state, or a combined driver state index. In a furtherembodiment, modifying control of one or more vehicle systems can bebased on a driver state, a level of a driver state, a driver stateindex, a combined driver state, a level of a combined driver state, thecombined driver state index, and/or a vehicular state. Accordingly,modifying the control of the one or more vehicle systems can includechanging at least one operating parameter of the one or more vehiclesystems based on a driver state, a level of a driver state, a driverstate index, a combined driver state, a level of a combined driverstate, the combined driver state index and/or a vehicular state. Theoperating parameter can be used to determine activation of a particularfunction of the one or more vehicle systems.

In some embodiments, modifying control of one or more vehicle systemscan include operating one or more vehicle systems based on a driverstate, a level of a driver state, a driver state index, a combineddriver state, a level of a combined driver state, the combined driverstate index, and/or a vehicular state. A control parameter can be usedto operate the one or more vehicle systems. In a one embodiment, thecontrol parameter is determined based on a driver state, a level of adriver state, a driver state index, a combined driver state, a level ofa combined driver state, the combined driver state index, and/or avehicular state.

Accordingly, the above described systems and methods provide multi-modalmonitoring and authentication of driver states. Utilizing such a system,provides a reliable and robust driver monitoring system that verifiesdriver states, provides a driver state (e.g., a combined driver state)based on multiple driver states using different types of monitoringsystems (e.g., multi-modal inputs) and modifies one or more vehiclesystems based on the driver state. In this way, behaviors and risks canbe assessed in multiple modes and modification of vehicle systems can becontrolled accurately. Exemplary types of operation, control, andmodification of one or more vehicle systems will now be described indetail. It is appreciated that the following examples are exemplary innature and other examples or combinations can be implemented.

A. Exemplary Operational Response of a Vehicle System to Driver State

In one embodiment, a response system can include provisions forcontrolling one or more vehicle systems to help wake a drowsy driverbased on the detected driver state. For example, a response system couldcontrol various systems to stimulate a driver in some way (visually,orally, or through movement, for example). A response system could alsochange ambient conditions in a motor vehicle to help wake the driver andthereby increase the driver's alertness.

FIGS. 63 and 64 illustrate a schematic view of a method of waking adriver by modifying the control of an electronic power steering system.FIGS. 63 and 64 will be described with reference to FIGS. 1A, 1B, 2, and3. Referring to FIG. 63, the driver 102 (e.g., of the motor vehicle 100)is drowsy. The response system 188 can detect that the driver 102 isdrowsy using any of the detection methods mentioned previously orthrough any other detection methods. During normal operation, the EPSsystem 132 functions to assist a driver in turning a touch steeringwheel 134. However, in some situations, it can be beneficial to reducethis assistance. For example, as seen in FIG. 64, by decreasing thepower steering assistance, the driver 102 must put more effort intoturning the touch steering wheel 134. This can have the effect of wakingup the driver 102, since the driver 102 must now apply a greater forceto turn the touch steering wheel 134.

FIG. 65 illustrates an embodiment of a process for controlling powersteering assistance according to the detected level of drowsiness for adriver. In some embodiments, some of the following steps could beaccomplished by a response system 188 of a motor vehicle. In some cases,some of the following steps can be accomplished by an ECU 106 of a motorvehicle. In other embodiments, some of the following steps could beaccomplished by other components of a motor vehicle, such as vehiclesystems 126. In still other embodiments, some of the following stepscould be accomplished by any combination of systems or components of thevehicle. It will be understood that in some embodiments one or more ofthe following steps can be optional. For purposes of reference, thefollowing method discusses components shown in FIGS. 1A, 1B through 3,including the response system 188.

In step 6502, the response system 188 can receive drowsinessinformation. In some cases, the drowsiness information includes whethera driver is in a normal state or a drowsy state. Moreover, in somecases, the drowsiness information could include a value indicating thelevel of drowsiness, for example on a scale of 1 to 10, with 1 being theleast drowsy and 10 being the drowsiest.

In step 6504, the response system 188 determines if the driver is drowsybased on the drowsiness information. If the driver is not drowsy, theresponse system 188 returns back to step 6502. If the driver is drowsy,the response system 188 proceeds to step 1506. In step 6506, steeringwheel information can be received. In some cases, the steering wheelinformation can be received from an EPS system 132. In other cases, thesteering wheel information can be received from a steering angle sensoror a steering torque sensor directly.

In step 6508, the response system 188 can determine if the driver isturning the steering wheel. If not, the response system 188 returns tostep 6502. If the driver is turning the steering wheel, the responsesystem 188 proceeds to step 6510 where the power steering assistance isdecreased. It will be understood that in some embodiments, the responsesystem 188 cannot check to see if the wheel is being turned beforedecreasing power steering assistance.

FIG. 66 illustrates an embodiment of a detailed process for controllingpower steering assistance to a driver according to a driver state index.In step 6602, the response system 188 can receive steering information.The steering information can include any type of information includingsteering angle, steering torque, rotational speed, motor speed as wellas any other steering information related to a steering system and/or apower steering assistance system. In step 6604, the response system 188can provide power steering assistance to a driver. In some cases, theresponse system 188 provides power steering assistance in response to adriver request (for example, when a driver turns on a power steeringfunction). In other cases, the response system 188 automaticallyprovides power steering assistance according to vehicle conditions orother information.

In step 6606, the response system 188 can determine the driver stateindex of a driver using any of the methods discussed above fordetermining a driver state index. Next, in step 6608, the responsesystem 188 can set a power steering status corresponding to the amountof steering assistance provided by the electronic power steering system.For example, in some cases, the power steering status is associated withtwo states, including a “low” state and a “standard” state. In the“standard” state, power steering assistance is applied at apredetermined level corresponding to an amount of power steeringassistance that improves drivability and helps increase the drivingcomfort of the user. In the “low” state, less steering assistance isprovided, which requires increased steering effort by a driver. Asindicated by look-up table 6610, the power steering status can beselected according to the driver state index. For example, if the driverstate index is 1 or 2 (corresponding to no drowsiness or slightdrowsiness), the power steering status is set to the standard state. If,however, the driver state index is 3 or 4 (corresponding to a drowsycondition of the driver), the power steering status is set to the lowstate. It will be understood that look-up table 6610 is only intended tobe exemplary and in other embodiments, the relationship between driverstate index and power steering status can vary in any manner.

Once the power steering status is set in step 6608, the response system188 proceeds to step 6612. In step 1528, the response system 188determines if the power steering status is set to low. If not, theresponse system 188 can return to step 6602 and continue operating powersteering assistance at the current level. However, if the responsesystem 188 determines that the power steering status is set to low, theresponse system 188 can proceed to step 6614. In step 6614, the responsesystem 188 can ramp down power steering assistance. For example, if thepower steering assistance is supplying a predetermined amount of torqueassistance, the power steering assistance can be varied to reduce theassisting torque. This requires the driver to increase steering effort.For a drowsy driver, the increased effort required to turn the steeringwheel can help increase his or her alertness and improve vehiclehandling.

In some cases, during step 6616, the response system 188 can provide awarning to the driver of the decreased power steering assistance. Forexample, in some cases, a dashboard light reading “power steering off”or “power steering decreased” could be turned on. In other cases, anavigation screen or other display screen associated with the vehiclecould display a message indicating the decreased power steeringassistance. In still other cases, an audible or haptic indicator couldbe used to alert the driver. This helps to inform the driver of thechange in power steering assistance so the driver does not becomeconcerned of a power steering failure.

FIGS. 67 and 68 illustrate schematic views of a method of helping towake a drowsy driver by automatically modifying the operation of aclimate control system. FIGS. 67 and 68 will be described with referenceto FIGS. 1A, 1B, 2, and 3. Referring to FIG. 67, a climate controlsystem 234 has been set to maintain a temperature of 75 degreesFahrenheit inside the cabin of the motor vehicle 100 by the driver 102.This is indicated on display screen 6702. As the response system 188detects that the driver 102 is becoming drowsy, the response system 188can automatically change the temperature of the climate control system234. As seen in FIG. 68, the response system 188 automatically adjuststhe temperature to 60 degrees Fahrenheit. As the temperature inside themotor vehicle 100 cools down, the driver 102 can become less drowsy.This helps the driver 102 to be more alert while driving. In otherembodiments, the temperature can be increased in order to make thedriver more alert.

FIG. 69 illustrates an embodiment of a process for helping to wake adriver by controlling the temperature in a vehicle. In some embodiments,some of the following steps could be accomplished by a response system188 of a motor vehicle. In some cases, some of the following steps canbe accomplished by an ECU 106 of a motor vehicle. In other embodiments,some of the following steps could be accomplished by other components ofa motor vehicle, such as vehicle systems 126. In still otherembodiments, some of the following steps could be accomplished by anycombination of systems or components of the vehicle. It will beunderstood that in some embodiments one or more of the following stepscan be optional. For purposes of reference, the following methoddiscusses components shown in FIGS. 1A, 1B through 3, including theresponse system 188.

In step 6902, the response system 188 may receive drowsinessinformation. In step 6904, the response system 188 determines if thedriver is drowsy. If the driver is not drowsy, the response system 188proceeds back to step 6902. If the driver is drowsy, the response system188 proceeds to step 6906. In step 6906, the response system 188automatically adjusts the cabin temperature. In some cases, the responsesystem 188 can lower the cabin temperature by engaging a fan orair-conditioner. However, in some other cases, the response system 188could increase the cabin temperature using a fan or heater. Moreover, itwill be understood that the embodiments are not limited to changingtemperature and in other embodiments other aspects of the in-cabinclimate could be changed, including airflow, humidity, pressure, orother ambient conditions. For example, in some cases, a response systemcould automatically increase the airflow into the cabin, which canstimulate the driver and help reduce drowsiness.

FIGS. 70 and 71 illustrate schematic views of methods of alerting adrowsy driver using visual, audible, and tactile feedback for a driver.FIGS. 70 and 71 will be described with reference to FIGS. 1A, 1B, 2, and3. Referring to FIG. 70, the driver 102 is drowsy as the motor vehicle100 is moving. Once the response system 188 detects this drowsy state,the response system 188 can activate one or more feedback mechanisms tohelp wake the driver 102. Referring to FIG. 71, three different methodsof waking a driver are shown. In particular, the response system 188 cancontrol one or more of the tactile devices 148. Examples of tactiledevices include vibrating devices (such as a vibrating seat or massagingseat) or devices whose surface properties can be modified (for example,by heating or cooling or by adjusting the rigidity of a surface). In oneembodiment, the response system 188 can operate the vehicle seat 168 toshake or vibrate. This can have the effect of waking the driver 102. Inother cases, steering wheel 134 could be made to vibrate or shake. Inaddition, in some cases, the response system 188 could activate one ormore lights or other visual indicators. For example, in one embodiment,a warning can be displayed on display screen 7002. In one example, thewarning can be “Wake!” and can include a brightly lit screen to catchthe driver's attention. In other cases, overhead lights or other visualindicators could be turned on to help wake the driver. In someembodiments, the response system 188 could generate various soundsthrough speakers 7004. For example, in some cases, the response system188 could activate a radio, CD player, MP3 player or other audio deviceto play music or other sounds through the speakers 7004. In other cases,the response system 188 could play various recordings stored in memory,such as voices that tell a driver to wake.

FIG. 72 illustrates an embodiment of a process for waking up a driverusing various visual, audible, and tactile stimuli. In some embodiments,some of the following steps could be accomplished by a response system188 of a motor vehicle. In some cases, some of the following steps canbe accomplished by an ECU 106 of a motor vehicle. In other embodiments,some of the following steps could be accomplished by other components ofa motor vehicle, such as vehicle systems 126. In still otherembodiments, some of the following steps could be accomplished by anycombination of systems or components of the vehicle. It will beunderstood that in some embodiments one or more of the following stepscan be optional. For purposes of reference, the following methoddiscusses components shown in FIGS. 1A, 1B through 3, including theresponse system 188.

In step 7202, the response system 188 can receive drowsinessinformation. In step 7204, the response system 188 determines if thedriver is drowsy. If the driver is not drowsy, the response system 188returns to step 7202. Otherwise, the response system 188 proceeds tostep 7206. In step 7206, the response system 188 can provide tactilestimuli to the driver. For example, the response system 188 couldcontrol a seat or other portion of the motor vehicle 100 to shake and/orvibrate (for example, a steering wheel). In other cases, the responsesystem 188 could vary the rigidity of a seat or other surface in themotor vehicle 100.

In step 7208, the response system 188 can turn on one or more lights orindicators. The lights could be any lights associated with the motorvehicle 100 including dashboard lights, roof lights or any other lights.In some cases, the response system 188 can provide a brightly litmessage or background on a display screen, such as a navigation systemdisplay screen or climate control display screen. In step 7210, theresponse system 188 can generate various sounds using speakers in themotor vehicle 100. The sounds could be spoken words, music, alarms, orany other kinds of sounds. Moreover, the volume level of the soundscould be chosen to ensure the driver is put in an alert state by thesounds, but not so loud as to cause great discomfort to the driver.

A response system can include provisions for controlling a seat beltsystem to help wake a driver. In some cases, a response system cancontrol an electronic pretensioning system for a seat belt to provide awarning pulse to a driver. FIGS. 73 and 74 illustrate schematic views ofan embodiment of a response system controlling an electronicpretensioning system for a seat belt. FIGS. 73 and 74 will be describedwith reference to FIGS. 1A, 1B, 2, and 3. Referring to FIGS. 73 and 74,as the driver 102 begins to feel drowsy, the response system 188 canautomatically control EPT system 236 to provide a warning pulse to thedriver 102. In particular, a seat belt 7302 can be initially loose asseen in FIG. 73, but as the driver 102 gets drowsy, the seat belt 7302is pulled taut against the driver 102 for a moment as seen in FIG. 74.This momentary tightening serves as a warning pulse that helps to wakethe driver 102.

FIG. 75 illustrates an embodiment of a process for controlling the EPTsystem 236. During step 7502, the response system 188 receivesdrowsiness information. During step 7504, the response system 188determines if the driver is drowsy. If the driver is not drowsy, theresponse system 188 returns to step 7502. If the driver is drowsy, theresponse system 188 proceeds to step 7506 where a warning pulse is sent.In particular, the seat belt can be tightened to help wake or alert thedriver.

In addition to controlling various vehicle systems to stimulate adriver, a motor vehicle can also include other provisions forcontrolling various vehicle systems (e.g., the vehicle systems in FIG.2) based on the driver state. The methods and systems for controllingvarious vehicle systems discussed herein are exemplary and it isunderstood that other modifications to other vehicle systems arecontemplated. For example, a motor vehicle can include provisions foradjusting various brake control systems according to the behavior of adriver. For example, a response system can modify the control ofantilock brakes, brake assist, brake prefill, as well as other brakingsystems when a driver is drowsy. This arrangement helps to increase theeffectiveness of the braking system in hazardous driving situations thatcan result when a driver is drowsy.

FIGS. 76 and 77 illustrate schematic views of the operation of anantilock braking system. FIGS. 76 and 77 will be described withreference to FIGS. 1A, 1B, 2, and 3. Referring to FIG. 76 when a driver102 is fully awake, the ABS system 204 can be associated with a firststopping distance 7602. In particular, for a particular initial speed7604, as a driver 102 depresses brake pedal 7606, the motor vehicle 100can travel to the first stopping distance 7602 before coming to acomplete stop. Thus, the first stopping distance 7602 can be the resultof various operating parameters of the ABS system 204.

Referring now to FIG. 77, as the driver 102 becomes drowsy, the responsesystem 188 can modify the control of the ABS system 204. In particular,in some cases, one or more operating parameters of the ABS system 204can be changed to decrease the stopping distance. In this case shown inFIG. 77, as the driver 102 depresses a brake pedal 7606, the motorvehicle 100 can travel to a second stopping distance 7608 before comingto a complete stop. In one embodiment, the second stopping distance 7608can be substantially shorter than the first stopping distance 7602. Inother words, the stopping distance can be decreased when the driver 102is drowsy. Since a drowsy driver can engage the brake pedal later due toa reduced awareness, the ability of the response system 188 to decreasethe stopping distance can help compensate for the reduced reaction timeof the driver. In another embodiment, if the vehicle is on a slipperysurface the reduction in stopping cannot occur and instead tactilefeedback can be applied through the brake pedal.

FIG. 78 illustrates an embodiment of a process for modifying the controlof an antilock braking system according to the behavior of a driver. Insome embodiments, some of the following steps could be accomplished by aresponse system 188 of a motor vehicle. In some cases, some of thefollowing steps can be accomplished by an ECU 106 of a motor vehicle. Inother embodiments, some of the following steps could be accomplished byother components of a motor vehicle, such as vehicle systems 126. Instill other embodiments, some of the following steps could beaccomplished by any combination of systems or components of the vehicle.It will be understood that in some embodiments one or more of thefollowing steps can be optional. For purposes of reference, thefollowing method discusses components shown in FIGS. 1A, 1 b through 3,including the response system 188.

In step 7802, the response system 188 can receive drowsinessinformation. In step 27804, the response system 188 can determine if thedriver is drowsy. If the driver is not drowsy, the response system 188returns to step 7802. If the driver is drowsy, the response system 188can proceed to step 7806. In step 7806, the response system 188 candetermine the current stopping distance. The current stopping distancecan be a function of the current vehicle speed, as well as otheroperating parameters including various parameters associated with thebrake system. In step 7808, the response system 188 can automaticallydecrease the stopping distance. This can be achieved by modifying one ormore operating parameters of the ABS system 204. For example, the brakeline pressure can be modified by controlling various valves, pumps,and/or motors within the ABS system 204. In a further embodiment, theidle stop function linked to the engine 104 and the braking systems canbe modified by turning the idle stop function OFF when the driver isdrowsy.

In some embodiments, a response system can automatically prefill one ormore brake lines in a motor vehicle in response to driver state. FIG. 79illustrates an embodiment of a process for controlling brake lines in amotor vehicle in response to driver state. In some embodiments, some ofthe following steps could be accomplished by a response system 188 of amotor vehicle. In some cases, some of the following steps can beaccomplished by an ECU 106 of a motor vehicle. In other embodiments,some of the following steps could be accomplished by other components ofa motor vehicle, such as vehicle systems 126. In still otherembodiments, some of the following steps could be accomplished by anycombination of systems or components of the vehicle. It will beunderstood that in some embodiments one or more of the following stepscan be optional. For purposes of reference, the following methoddiscusses components shown in FIGS. 1A, 1B through 3, including theresponse system 188.

In step 7902, the response system 188 can receive drowsinessinformation. In step 7904, the response system 188 can determine if thedriver is drowsy. If the driver is not drowsy, the response system 188can return to step 7902. If the driver is drowsy, the response system188 can automatically prefill the brake lines with brake fluid in step7906. For example, the response system 188 can use the automatic brakeprefill system 208. In some cases, this can help increase brakingresponse if a hazardous condition arises while the driver is drowsy. Itwill be understood that any number of brake lines could be prefilledduring step 7906. Moreover, any provisions known in the art forprefilling brake lines could be used including any pumps, valves, motorsor other devices needed to supply brake fluid automatically to brakelines.

Some vehicles can be equipped with brake assist systems that help reducethe amount of force a driver must apply to engage the brakes. Thesesystems can be activated for older drivers or any other drivers who canneed assistance with braking. In some cases, a response system couldutilize the brake assist systems when a driver is drowsy, since a drowsydriver may not be able to apply the necessary force to the brake pedalfor stopping a vehicle quickly.

FIG. 80 illustrates an embodiment of a method for controlling automaticbrake assist in response to driver state. In step 8002, the responsesystem 188 can receive drowsiness information. In step 8004, theresponse system 188 can determine if the driver is drowsy. If the driveris not drowsy, the response system 188 proceeds back to step 8002. Ifthe driver is drowsy, the response system 188 can determine if the brakeassist system 206 is already on in step 8006. If the brake assist system206 is already on, the response system 188 can return to step 8002. Ifthe brake assist system 206 is not currently active, the response system188 can turn on the brake assist system 206 in step 8008. Thisarrangement allows for braking assistance to a drowsy driver, since thedriver may not have sufficient ability to supply the necessary brakingforce in the event that the motor vehicle 100 must be stopped quickly.

In some embodiments, a response system could modify the degree ofassistance in a brake assist system. For example, a brake assist systemcan operate under normal conditions with a predetermined activationthreshold. The activation threshold can be associated with the rate ofchange of the master cylinder brake pressure. If the rate of change ofthe master cylinder brake pressure exceeds the activation threshold,brake assist can be activated. However, when a driver is drowsy, thebrake assist system can modify the activation threshold so that brakeassist is activated sooner. In some cases, the activation thresholdcould vary according to the degree of drowsiness. For example, if thedriver is only slightly drowsy, the activation threshold can be higherthan when the driver is extremely drowsy.

FIG. 81 illustrates an embodiment of a detailed process for controllingautomatic brake assist in response to driver state. In particular, FIG.81 illustrates a method in which brake assist is modified according tothe driver state index of the driver. In step 8102, the response system188 can receive braking information. Braking information can includeinformation from any sensors and/or vehicle systems. In step 8104, theresponse system 188 can determine if a brake pedal is depressed. In somecases, the response system 188 can receive information that a brakeswitch has been applied to determine if the driver is currently braking.In other cases, any other vehicle information can be monitored todetermine if the brakes are being applied. In step 8106, the responsesystem 188 can measure the rate of brake pressure increase. In otherwords, the response system 188 determines how fast the brake pressure isincreasing, or how “hard” the brake pedal is being depressed. In step8108, the response system 188 sets an activation threshold. Theactivation threshold corresponds to a threshold for the rate of brakepressure increase. Details of this step are discussed in detail below.

In step 8110, the response system 188 determines if the rate of brakepressure increase exceeds the activation threshold. If not, the responsesystem 188 proceeds back to step 8102. Otherwise, the response system188 proceeds to step 8112. In step 8112, the response system 188activates a modulator pump and/or valves to automatically increase thebrake pressure. In other words, in step 8112, the response system 188activates brake assist. This allows for an increase in the amount ofbraking force applied at the wheels.

FIG. 82 illustrates an embodiment of a process of selecting theactivation threshold discussed above. In some embodiments, the processshown in FIG. 82 corresponds to step 8108 of FIG. 82. In step 8202, theresponse system 188 can receive the brake pressure rate and vehiclespeed as well as any other operating information. The brake pressurerate and vehicle speed correspond to current vehicle conditions that canbe used for determining an activation threshold under normal operatingconditions. In step 8204, an initial threshold setting can be determinedaccording to the vehicle operating conditions.

In order to accommodate changes in brake assist due to drowsiness, theinitial threshold setting can be modified according to the state of thedriver. In step 8206, the response system 188 determines the driverstate index of the driver using any method discussed above. Next, instep 8208, the response system 188 determines a brake assistcoefficient. As seen in look-up table 8210, the brake assist coefficientcan vary between 0% and 25% according to the driver state index.Moreover, the brake assist coefficient generally increases as the driverstate index increases. In step 8212, the activation threshold isselected according to the initial threshold setting and the brake assistcoefficient. If the brake assist coefficient has a value of 0%, theactivation threshold is just equal to the initial threshold setting.However, if the brake assist coefficient has a value of 25%, theactivation threshold can be modified by up to 25% in order to increasethe sensitivity of the brake assist when the driver is drowsy. In somecases, the activation threshold can be increased by up to 25% (or anyother amount corresponding to the brake assist coefficient). In othercases, the activation threshold can be decreased by up to 25% (or anyother amount corresponding to the brake assist coefficient).

A motor vehicle can include provisions for increasing vehicle stabilitywhen a driver is drowsy. In some cases, a response system can modify theoperation of an electronic stability control system. For example, insome cases, a response system could ensure that a detected yaw rate anda steering yaw rate (the yaw rate estimated from steering information)are very close to one another. This can help enhance steering precisionand reduce the likelihood of hazardous driving conditions while thedriver is drowsy.

FIGS. 83 and 84 are schematic views of an embodiment of the motorvehicle 100 turning around a curve in roadway 8300. FIGS. 83 and 84 willbe described with reference to FIGS. 1A, 1B, 2, and 3. Referring to FIG.83, the driver 102 is wide-awake and turning a steering wheel 134. Alsoshown in FIG. 83 are a driver intended path 8302 and an actual vehiclepath 8304. The driver intended path can be determined from steeringwheel information, yaw rate information, lateral g information, as wellas other kinds of operating information. The driver intended pathrepresents the ideal path of the vehicle, given the steering input fromthe driver. However, due to variations in road traction as well as otherconditions, the actual vehicle path can vary slightly from the driverintended path. Referring to FIG. 84, as the driver 102 gets drowsy, theresponse system 188 modifies the operation of the electronic stabilitycontrol system 202. In particular, the ESC system 202 is modified sothat the actual vehicle path 8402 is closer to the driver intended path8404. This helps to minimize the difference between the driver intendedpath and the actual vehicle path when the driver is drowsy, which canhelp improve driving precision.

FIG. 85 illustrates an embodiment of a process for controlling anelectronic vehicle stability system according to driver state. In someembodiments, some of the following steps could be accomplished by aresponse system 188 of a motor vehicle. In some cases, some of thefollowing steps can be accomplished by an ECU 106 of a motor vehicle. Inother embodiments, some of the following steps could be accomplished byother components of a motor vehicle, such as vehicle systems 126. Instill other embodiments, some of the following steps could beaccomplished by any combination of systems or components of the vehicle.It will be understood that in some embodiments one or more of thefollowing steps can be optional. For purposes of reference, thefollowing method discusses components shown in FIGS. 1A, 1B through 3,including the response system 188.

In step 8502, the response system 188 can receive drowsinessinformation. In step 8504, the response system 188 determines if thedriver is drowsy. If the driver is not drowsy, the response system 188can return to step 8502. Otherwise, the response system 188 receives yawrate information in step 8506. The yaw rate information could bereceived from a yaw rate sensor in some cases. In step 8508, theresponse system 188 receives steering information. This could include,for example, the steering wheel angle received from a steering anglesensor. In step 8510, the response system 188 determines the steeringyaw rate using the steering information. In some cases, additionaloperating information could be used to determine the steering yaw rate.In step 8512, the response system 188 can reduce the allowable errorbetween the measured yaw rate and the steering yaw rate. In other words,the response system 188 helps minimize the difference between the driverintended path and the actual vehicle path.

In order to reduce the allowable error between the yaw rate and thesteering yaw rate, the response system 188 can apply braking to one ormore brakes of the motor vehicle 100 in order to maintain the motorvehicle 100 close to the driver intended path. Examples of maintaining avehicle close to a driver intended path can be found in Ellis et al.,U.S. Pat. No. 8,426,257, filed Mar. 17, 2010, the entirety of which ishereby incorporated by reference.

FIG. 86 illustrates an embodiment of a process for controlling anelectronic stability control system in response to driver state. Inparticular, FIG. 86 illustrates an embodiment in which the operation ofthe electronic stability control system is modified according to thedriver state index of the driver. In step 8602, the response system 188receives operating information. This information can include anyoperating information such as yaw rate, wheel speed, steering angles, aswell as other information used by an electronic stability controlsystem. In step 8604, the response system 188 can determine if thevehicle behavior is stable. In particular, in step 8606, the responsesystem 188 measures the stability error of steering associated withunder-steering or over-steering. In some cases, the stability isdetermined by comparing the actual path of the vehicle with the driverintended path.

In step 8608, the response system 188 sets an activation thresholdassociated with the electronic stability control system. The activationthreshold can be associated with a predetermined stability error. Instep 8610, the response system 188 determines if the stability errorexceeds the activation threshold. If not, the response system 188 canreturn to step 8602. Otherwise, the response system 188 can proceed tostep 8612. In step 8612, the response system 188 applies individualwheel brake control in order to increase vehicle stability. In someembodiments, the response system 188 could also control the engine toapply engine braking or modify cylinder operation in order to helpstabilize the vehicle.

In some cases, in step 8614, the response system 188 can activate awarning indicator. The warning indicator could be any dashboard light ormessage displayed on a navigation screen or other video screen. Thewarning indicator helps to alert a driver that the electronic stabilitycontrol system has been activated. In some cases, the warning could bean audible warning and/or a haptic warning.

FIG. 87 illustrates an embodiment of a process for setting theactivation threshold used in the previous method. In step 8702, theresponse system 188 receives vehicle operating information. For example,the vehicle operating information can include wheel speed information,road surface conditions (such as curvature, friction coefficients,etc.), vehicle speed, steering angle, yaw rate, as well as otheroperating information. In step 8704, the response system 188 determinesan initial threshold setting according to the operating informationreceived in step 8702. In step 8706, the response system 188 determinesthe driver state index of the driver.

In step 8708, the response system 188 determines a stability controlcoefficient. As seen in look-up table 8710, the stability controlcoefficient can be determined from the driver state index. In oneexample, the stability control coefficient ranges from 0% to 25%.Moreover, the stability control coefficient generally increases with thedriver state index. For example, if the driver state index is 1, thestability control coefficient is 0%. If the driver state index is 4, thestability control coefficient is 25%. It will be understood that theseranges for the stability control coefficient are only intended to beexemplary and in other cases, the stability control coefficient couldvary in any other manner as a function of the driver state index.

In step 8712, the response system 188 can set the activation thresholdusing the initial threshold setting and the stability controlcoefficient. For example, if the stability control coefficient has avalue of 25%, the activation threshold can be up to 25% larger than theinitial threshold setting. In other cases, the activation threshold canbe up to 25% smaller than the initial threshold setting. In other words,the activation threshold can be increased or decreased from the initialthreshold setting in proportion to the value of the stability controlcoefficient. This arrangement helps to increase the sensitivity of theelectronic stability control system by modifying the activationthreshold in proportion to the state of the driver.

FIG. 88 illustrates a schematic view of the motor vehicle 100 equippedwith a collision warning system 218. The collision warning system 218can function to provide warnings about potential collisions to a driver.For purposes of clarity, the term “host vehicle” as used throughout thisdetailed description and in the claims refers to any vehicle including aresponse system while the term “target vehicle” refers to any vehiclemonitored by, or otherwise in communication with, a host vehicle. In thecurrent embodiment, for example, the motor vehicle 100 can be a hostvehicle. In this example, as the motor vehicle 100 approaches anintersection 8800 while a target vehicle 8802 passes through theintersection 8800, the collision warning system 218 can provide awarning alert 8804 on a display screen 8806. Further examples ofcollision warning systems are disclosed in Mochizuki, U.S. Pat. No.8,558,718, filed Sep. 20, 2010, and Mochizuki et al., U.S. Pat. No.8,587,418, filed Jul. 28, 2010, the entirety of both being herebyincorporated by reference.

FIG. 89 illustrates an embodiment of a process for modifying a collisionwarning system according to driver state. In some embodiments, some ofthe following steps could be accomplished by a response system 188 of amotor vehicle. In some cases, some of the following steps can beaccomplished by an ECU 106 of a motor vehicle. In other embodiments,some of the following steps could be accomplished by other components ofa motor vehicle, such as vehicle systems 126. In still otherembodiments, some of the following steps could be accomplished by anycombination of systems or components of the vehicle. It will beunderstood that in some embodiments one or more of the following stepscan be optional. For purposes of reference, the following methoddiscusses components shown in FIGS. 1A, 1B through 3, including theresponse system 188.

In step 8902, the response system 188 may receive drowsinessinformation. In step 8904, the response system 188 can determine if thedriver is drowsy. If the driver is not drowsy, the response system 188can proceed back to step 8902. Otherwise, the response system 188 canproceed to step 8906. In step 8906, the response system 188 can modifythe operation of a collision warning system so that the driver is warnedearlier about potential collisions. For example, if the collisionwarning system was initially set to warn a driver about a potentialcollision if the distance to the collision point is less than 25 meters,the response system 188 could modify the system to warn the driver ifthe distance to the collision point is less than 50 meters.

FIG. 90 illustrates an embodiment of a process for modifying a collisionwarning system according to driver state. In some embodiments, some ofthe following steps could be accomplished by a response system 188 of amotor vehicle. In some cases, some of the following steps can beaccomplished by an ECU 106 of a motor vehicle. In other embodiments,some of the following steps could be accomplished by other components ofa motor vehicle, such as vehicle systems 126. In still otherembodiments, some of the following steps could be accomplished by anycombination of systems or components of the vehicle. It will beunderstood that in some embodiments one or more of the following stepscan be optional. For purposes of reference, the following methoddiscusses components shown in FIGS. 1A, 1B through 3, including theresponse system 188.

In step 9002, the collision warning system 218 can retrieve the heading,position, and speed of an approaching vehicle. In some cases, thisinformation could be received from the approaching vehicle through awireless network, such as a DSRC network. In other cases, thisinformation could be remotely sensed using radar, lidar or other remotesensing devices.

In step 9004, the collision warning system 218 can estimate a vehiclecollision point. The vehicle collision point is the location of apotential collision between the motor vehicle 100 and the approachingvehicle, which could be traveling in any direction relative to the motorvehicle 100. In some cases, in step 9004, the collision warning system218 can use information about the position, heading, and speed of themotor vehicle 100 to calculate the vehicle collision point. In someembodiments, this information could be received from a GPS receiver thatis in communication with the collision warning system 218 or theresponse system 188. In other embodiments, the vehicle speed could bereceived from a vehicle speed sensor.

In step 9006, the collision warning system 218 can calculate thedistance and/or time to the vehicle collision point. In particular, todetermine the distance, the collision warning system 218 can calculatethe difference between the vehicle collision point and the currentlocation of the motor vehicle 100. Likewise, to determine the time tothe collision warning system 218 could calculate the amount of time itwill take to reach the vehicle collision point.

In step 9008, the collision warning system 218 can receive drowsinessinformation from the response system 188, or any other system orcomponents. In step 9010, the collision warning system 218 can determineif the driver is drowsy. If the driver is not drowsy, the collisionwarning system 218 can proceed to step 9012, where a first thresholdparameter is retrieved. If the driver is drowsy, the collision warningsystem 218 can proceed to step 9014, where a second threshold distanceis retrieved. The first threshold parameter and the second thresholdparameter could be either time thresholds or distance thresholds,according to whether the time to collision or distance to collision wasdetermined during step 9006. In some cases, where both time and distanceto the collision point are used, the first threshold parameter and thesecond threshold parameter can each comprise both a distance thresholdand a time threshold. Moreover, it will be understood that the firstthreshold parameter and the second threshold parameter can besubstantially different thresholds in order to provide a differentoperating configuration for the collision warning system 218 accordingto whether the driver is drowsy or not drowsy. Following both step 9012and 9014, collision warning system 218 proceeds to step 9016. In step9016, the collision warning system 218 determines if the currentdistance and/or time to the collision point is less than the thresholdparameter selected during the previous step (either the first thresholdparameter or the second threshold parameter).

The first threshold parameter and the second threshold parameter couldhave any values. In some cases, the first threshold parameter can beless than the second threshold parameter. In particular, if the driveris drowsy, it can be beneficial to use a lower threshold parameter,since this corresponds to warning a driver earlier about a potentialcollision. If the current distance or time is less than the thresholddistance or time (the threshold parameter), the collision warning system218 can warn the driver in step 9018. Otherwise, the collision warningsystem 218 may not warn the driver in step 9020.

A response system can include provisions for modifying the operation ofan automatic cruise control system according to driver state. In someembodiments, a response system can change the headway distanceassociated with an automatic cruise control system. In some cases, theheadway distance is the closest distance a motor vehicle can get to apreceding vehicle. If the automatic cruise control system detects thatthe motor vehicle is closer than the headway distance, the system canwarn the driver and/or automatically slow the vehicle to increase theheadway distance.

FIGS. 91 and 92 illustrate schematic views of the motor vehicle 100cruising behind a preceding vehicle 9102. In this situation, theautomatic cruise control system 216 is operating to automaticallymaintain a predetermined headway distance behind the preceding vehicle9102. When the driver 102 is awake, automatic cruise control system 216uses a first headway distance 9104, as seen in FIG. 91. In other words,the automatic cruise control system 216 automatically prevents the motorvehicle 100 from getting closer than the first headway distance 9104 tothe preceding vehicle 9102. As the driver 102 becomes drowsy, as seen inFIG. 92, the response system 188 can modify the operation of theautomatic cruise control system 216 so that the automatic cruise controlsystem 216 increases the headway distance to a second headway distance9106. The second headway distance 9106 can be substantially larger thanthe first headway distance 9104, since the reaction time of the driver102 can be reduced when the driver 102 is drowsy.

FIG. 93 illustrates an embodiment of a method of modifying the controlof an automatic cruise control system according to driver state. In someembodiments, some of the following steps could be accomplished by aresponse system 188 of a motor vehicle. In some cases, some of thefollowing steps can be accomplished by an ECU 106 of a motor vehicle. Inother embodiments, some of the following steps could be accomplished byother components of a motor vehicle, such as vehicle systems 126. Instill other embodiments, some of the following steps could beaccomplished by any combination of systems or components of the vehicle.It will be understood that in some embodiments one or more of thefollowing steps can be optional. For purposes of reference, thefollowing method discusses components shown in FIGS. 1A, 1B through 3,including the response system 188.

In step 9302, the response system 188 can receive drowsinessinformation. In step 9304, the response system 188 can determine if thedriver is drowsy. If the driver is not drowsy, the response system 188can return to step 9302. If the driver is drowsy, the response system188 can proceed to step 9306. In step 9306, the response system 188 candetermine if automatic cruise control is being used. If not, theresponse system 188 can return back to step 9302. If automatic cruisecontrol is being used, the response system 188 can proceed to step 9308.In step 9308, the response system 188 can retrieve the current headwaydistance for automatic cruise control. In step 9310, the response system188 can increase the headway distance. With this arrangement, theresponse system 188 can help increase the distance between the motorvehicle 100 and other vehicles when a driver is drowsy to reduce thechances of a hazardous driving situation while the driver is drowsy.

FIG. 94 illustrates an embodiment of a process for controlling automaticcruise control in response to driver state. This embodiment could alsoapply to normal cruise control systems. In particular, FIG. 94illustrates an embodiment of a process where the operation of anautomatic cruise control system is varied in response to the driverstate index of a driver. In step 9402, the response system 188 candetermine that the automatic cruise control function is turned on. Thiscan occur when a driver selects to turn on cruise control. In step 9404,the response system 188 can determine the driver state index of thedriver using any method discussed above as well as any method known inthe art. In step 9406, the response system 188 can set the automaticcruise control status based on the driver state index of the driver. Forexample, look-up table 9408 indicates that the automatic cruise controlstatus is set to on for driver state indexes of 1, 2, and 3. Also, theautomatic cruise control status is set to off for driver state index of4. In other embodiments, the automatic cruise control status can be setaccording to driver state index in any other manner.

In step 9410, the response system 188 determines if the automatic cruisecontrol status is ON. If so, the response system 188 proceeds to step9412. Otherwise, if the status is OFF, the response system 188 proceedsto step 9414. In step 9414, the response system 188 ramps down controlof automatic cruise control. For example, in some cases the responsesystem 188 can slow down the vehicle gradually to a predetermined speed.In step 9416, the response system 188 can turn off automatic cruisecontrol. In some cases, in step 9418, the response system 188 can informthe driver that automatic cruise control has been deactivated using adashboard warning light or message displayed on a screen of some kind.In other cases, the response system 188 could provide an audible warningthat automatic cruise control has been deactivated. In still othercases, a haptic warning could be used.

If the automatic cruise control status is determined to be on duringstep 9410, the response system 188 can set the automatic cruise controldistance setting in step 9412. For example, look-up table 9420 providesone possible configuration for a look-up table relating the driver stateindex to a distance setting. In this case, a driver state index of 1corresponds to a first distance, a driver state index of 2 correspondsto a second distance, and a driver state index of 3 corresponds to athird distance. Each distance can have a substantially different value.In some cases, the value of each headway distance can increase as thedriver state index increases in order to provide more headway room fordrivers who are drowsy or otherwise inattentive. In step 9422, theresponse system 188 can operate automatic cruise control using thedistance setting determined during step 9412.

A response system can include provisions for automatically reducing acruising speed in a cruise control system based on driver monitoringinformation. FIG. 95 illustrates an embodiment of a method forcontrolling a cruising speed. In some embodiments, some of the followingsteps could be accomplished by a response system 188 of a motor vehicle.In some cases, some of the following steps can be accomplished by an ECU106 of a motor vehicle. In other embodiments, some of the followingsteps could be accomplished by other components of a motor vehicle, suchas vehicle systems 126. In still other embodiments, some of thefollowing steps could be accomplished by any combination of systems orcomponents of the vehicle. It will be understood that in someembodiments one or more of the following steps can be optional. Forpurposes of reference, the following method discusses components shownin FIGS. 1A, 1B through 3, including the response system 188.

In step 9502, the response system 188 can receive drowsinessinformation. In step 9504, the response system 188 can determine if thedriver is drowsy. If the driver is not drowsy, the response system 188returns to step 9502, otherwise the response system 188 proceeds to step9506. In step 9506, the response system 188 determines if cruise controlis operating. If not, the response system 188 returns back to step 9502.If cruise control is operating, the response system 188 determines thecurrent cruising speed in step 9508. In step 9510, the response system188 retrieves a predetermined percentage. The predetermined percentagecould have any value between 0% and 100%. In step 9512, the responsesystem 188 can reduce the cruising speed by the predeterminedpercentage. For example, if the motor vehicle 100 is cruising at 60 mphand the predetermined percentage is 50%, the cruising speed can bereduced to 30 mph. In other embodiments, the cruising speed could bereduced by a predetermined amount, such as by 20 mph or 30 mph. In stillother embodiments, the predetermined percentage could be selected from arange of percentages according to the driver body index. For example, ifthe driver is only slightly drowsy, the predetermined percentage couldbe smaller than the percentage used when the driver is very drowsy.Using this arrangement, the response system 188 can automatically reducethe speed of the motor vehicle 100, since slowing the vehicle can reducethe potential risks posed by a drowsy driver.

FIG. 96 illustrates an embodiment of a process for controlling a lowspeed follow system 212 in response to driver state. In step 9602, theresponse system 188 can determine if the low speed follow system is on.“Low speed follow” refers to any system that is used for automaticallyfollowing a preceding vehicle at low speeds.

In step 9604, the response system 188 can determine the driver stateindex of the driver. Next, in step 9606, the response system 188 can setthe low speed follow status based on the driver state index of thedriver. For example, look-up table 9610 shows an exemplary relationshipbetween driver state index and the low speed follow status. Inparticular, the low speed follow status varies between an “on” state andan “off” state. For low driver state index (driver state indexes of 1 or2) the low speed follow status can be set to “ON.” For high driver stateindex (driver state indexes of 3 or 4) the low speed follow status canbe set to “OFF.” It will be understood that the relationship betweendriver state index and low speed follow status shown here is onlyexemplary and in other embodiments the relationship could vary in anyother manner.

In step 9612, the response system 188 determines if the low speed followstatus is ON or OFF. If the low speed follow status is ON, the responsesystem 188 returns to step 9602. Otherwise, the response system 188proceeds to step 9614 when the low speed follow status is off. In step9614, the response system 188 can ramp down control of the low speedfollow function. For example, the low speed follow system 212 cangradually increase the headway distance with the preceding vehicle untilthe system is shut down in step 9616. By automatically turning of lowspeed follow when a driver is drowsy, the response system 188 can helpincrease driver attention and awareness since the driver must put moreeffort into driving the vehicle.

In some cases, in step 9618, the response system 188 can inform thedriver that low speed follow has been deactivated using a dashboardwarning light or message displayed on a screen of some kind. In othercases, the response system 188 could provide an audible warning that lowspeed follow has been deactivated.

A response system can include provisions for modifying the operation ofa lane departure warning system 222, which helps alert a driver if themotor vehicle is unintentionally leaving the current lane. In somecases, a response system could modify when the lane departure warningsystem 222 alerts a driver. For example, the lane keep departure warningsystem could warn the driver before the vehicle crosses a lane boundaryline, rather than waiting until the vehicle has already crossed the laneboundary line.

FIGS. 97 and 98 illustrate schematic views of an embodiment of a methodof modifying the operation of a lane departure warning system 222. Themotor vehicle 100 travels on a roadway 9700. Under circumstances where adriver 102 is fully alert (see FIG. 97), the lane departure warningsystem 222 can wait until the motor vehicle 100 crosses a lane boundaryline 9702 before providing a warning 9704. However, in circumstanceswhere the driver 102 is drowsy (see FIG. 98), the lane departure warningsystem 222 can provide the warning 9704 just prior to the moment whenthe motor vehicle 100 crosses the lane boundary line 9702. In otherwords, the lane departure warning system 222 warns the driver 102earlier when the driver 102 is drowsy. This can help improve thelikelihood that the driver 102 stays inside the current lane.

FIG. 99 illustrates an embodiment of a process of operating a lanedeparture warning system 222 in response to driver state. In someembodiments, some of the following steps could be accomplished by aresponse system 188 of a motor vehicle. In some cases, some of thefollowing steps can be accomplished by an ECU 106 of a motor vehicle. Inother embodiments, some of the following steps could be accomplished byother components of a motor vehicle, such as vehicle systems 126. Instill other embodiments, some of the following steps could beaccomplished by any combination of systems or components of the vehicle.It will be understood that in some embodiments one or more of thefollowing steps can be optional. For purposes of reference, thefollowing method discusses components shown in FIGS. 1A, 1B through 3,including the response system 188.

In step 9902, the response system 188 can retrieve drowsinessinformation. In step 9904, the response system 188 can determine if thedriver is drowsy. If the driver is not drowsy, the response system 188proceeds back to step 9902. Otherwise, the response system 188 proceedsto step 9906. In step 9906, the response system 188 can modify theoperation of lane departure warning system 222 so that the driver iswarned earlier about potential lane departures.

FIG. 100 illustrates an embodiment of a process for operating a lanedeparture warning system 222 in response to driver state. In particular,FIG. 100 illustrates an embodiment of a process where the operation of alane departure warning system 222 is modified in response to the driverstate index of a driver. In step 10002, the response system 188 receivesroadway information. The roadway information can include road size,shape as well as the locations of any road markings or lines. In step10004, the response system 188 can determine the vehicle positionrelative to the road. In step 10006, the response system 188 cancalculate the time to lane crossing. This can be determined from vehicleposition, vehicle turning information, and lane location information.

In step 10008, the response system 188 can set the road crossingthreshold. The road crossing threshold can be a time associated with thetime to lane crossing. In step 10010, the response system 188 determinesif the time to lane crossing exceeds the road crossing threshold. Ifnot, the response system 188 proceeds back to step 10002. Otherwise, theresponse system 188 proceeds to step 10012 where a warning indicator isilluminated indicating that the vehicle is crossing a lane. In othercases, audible or haptic warnings could also be provided. If the vehiclecontinues exiting, the lane a steering effort correction can be appliedin step 10014.

FIG. 101 illustrates an embodiment of a process for setting the roadcrossing threshold. In step 10102, the response system 188 determines aminimum reaction time for vehicle recovery. In some cases, the minimumreaction time is associated with the minimum amount of time for avehicle to avoid a lane crossing once a driver becomes aware of thepotential lane crossing. In step 10104, the response system 188 canreceive vehicle operating information. Vehicle operating informationcould include roadway information as well as information related to thelocation of the vehicle within the roadway.

In step 10106, the response system 188 determines an initial thresholdsetting from the minimum reaction time and the vehicle operatinginformation. In step 10108, the response system 188 determines the bodyindex state of the driver. In step 10110, the response system 188determines a lane departure warning coefficient according to the driverstate index. An exemplary look-up table 10112 includes a range ofcoefficient values between 0% and 25% as a function of the driver stateindex. Finally, in step 10114, the response system 188 can set the roadcrossing threshold according to the lane departure warning coefficientand the initial threshold setting.

In addition to providing earlier warnings to a driver through a lanedeparture warning system, the response system 188 can also modify theoperation of a lane keep assist system, which can also provide warningsas well as driving assistance in order to maintain a vehicle in apredetermined lane.

FIG. 102 illustrates an embodiment of a process of operating a lane keepassist system in response to driver state. In particular, FIG. 102illustrates a method where the operation of a lane keep assist system ismodified in response to the driver state index of a driver. In step10202, the response system 188 can receive operating information. Forexample, in some cases the response system 188 can receive roadwayinformation related to the size and/or shape of a roadway, as well asthe location of various lines on the roadway. In step 10204, theresponse system 188 determines the location of the road center and thewidth of the road. This can be determined using sensed information, suchas optical information of the roadway, stored information including mapbased information, or a combination of sensed and stored information. Instep 10206, the response system 188 can determine the vehicle positionrelative to the road.

In step 10208, the response system 188 can determine the deviation ofthe vehicle path from the road center. In step 10210, the responsesystem 188 can learn the driver's centering habits. For example, alertdrivers generally adjust the steering wheel constantly in attempt tomaintain the car in the center of a lane. In some cases, the centeringhabits of a driver can be detected by the response system 188 andlearned. Any machine learning method or pattern recognition algorithmcould be used to determine the driver's centering habits.

In step 10212, the response system 188 can determine if the vehicle isdeviating from the center of the road. If not, the response system 188proceeds back to step 10202. If the vehicle is deviating, the responsesystem 188 proceeds to step 10214. In step 10214, the response system188 can determine the driver state index of the driver. Next, in step10216, the response system 188 can set the lane keep assist status usingthe driver state index. For example, a look-up table 10218 is an exampleof a relationship between driver state index and lane keep assiststatus. In particular, the lane keep assist status is set to a standardstate for low driver state index (indexes 1 or 2) and is set to a lowstate for a higher driver state index (indexes 3 or 4). In otherembodiments, any other relationship between driver state index and lanekeep assist status can be used.

In step 10220, the response system 188 can check the lane keep assiststatus. If the lane keep assist status is standard, the response system188 proceeds to step 10222 where standard steering effort correctionsare applied to help maintain the vehicle in the lane. If, however, theresponse system 188 determines that the lane keep assist status is lowin step 10220, the response system 188 can proceed to step 10224. Instep 10224, the response system 188 determines if the road is curved. Ifnot, the response system 188 proceeds to step 10226 to illuminate a lanekeep assist warning so the driver knows the vehicle is deviating fromthe lane. If, in step 10224, the response system 188 determines the roadis curved, the response system 188 proceeds to step 10228. In step10228, the response system 188 determines if the driver's hands are onthe steering wheel. If so, the response system 188 proceeds to step10230 where the process ends. Otherwise, the response system 188proceeds to step 10226.

This arrangement allows the response system 188 to modify the operationof the lane keep assist system in response to driver state. Inparticular, the lane keep assist system can only help steer the vehicleautomatically when the driver state is alert (low driver state index).Otherwise, if the driver is drowsy or very drowsy (higher driver stateindex), the response system 188 can control the lane keep assist systemto only provide warnings of lane deviation without providing steeringassistance. This can help increase the alertness of the driver when heor she is drowsy.

A response system can include provisions for modifying the control of ablind spot indicator system when a driver is drowsy. For example, insome cases, a response system could increase the detection area. Inother cases, the response system could control the monitoring system todeliver warnings earlier (i.e., when an approaching vehicle is furtheraway).

FIGS. 103 and 104 illustrate schematic views of an embodiment of theoperation of a blind spot indicator system. In this embodiment, themotor vehicle 100 is traveling on roadway 10302. The blind spotindicator system 224 (see FIG. 2) can be used to monitor any objectstraveling within a blind spot monitoring zone 10304. For example, in thecurrent embodiment, the blind spot indicator system 224 can determinethat no object is inside of the blind spot monitoring zone 10304. Inparticular, a target vehicle 10306 is just outside of the blind spotmonitoring zone 1304. In this case, n103o alert is sent to the driver.

In FIG. 103, the driver 102 is shown as fully alert. In this alertstate, the blind spot monitoring zone is set according to predeterminedsettings and/or vehicle operating information. However, as seen in FIG.104, as the driver 102 becomes drowsy, the response system 188 canmodify the operation of the blind spot indicator system 224. Forexample, in one embodiment, the response system 188 can increase thesize of the blind spot monitoring zone 10304. As seen in FIG. 104, underthese modified conditions the target vehicle 10306 is now travelinginside of the blind spot monitoring zone 10304. Therefore, in thissituation the driver 102 is alerted (e.g., alert 10308) to the presenceof the target vehicle 10306.

FIG. 105 illustrates an embodiment of a process of operating a blindspot indicator system in response to driver state. In some embodiments,some of the following steps could be accomplished by a response system188 of a motor vehicle. In some cases, some of the following steps canbe accomplished by an ECU 106 of a motor vehicle. In other embodiments,some of the following steps could be accomplished by other components ofa motor vehicle, such as vehicle systems 126. In still otherembodiments, some of the following steps could be accomplished by anycombination of systems or components of the vehicle. It will beunderstood that in some embodiments one or more of the following stepscan be optional. For purposes of reference, the following methoddiscusses components shown in FIGS. 1A, 1B through 3, including theresponse system 188.

In step 10502, the response system 188 can receive drowsinessinformation. In step 10504, the response system 188 determines if thedriver is drowsy. If the driver is not drowsy, the response system 188returns back to step 10502. If the driver is drowsy, the response system188 proceeds to step 10506. In step 10506, response system 188 canincrease the blind spot detection area. For example, if the initialblind spot detection area is associated with the region of the vehiclebetween the passenger side mirror about 3-5 meters behind the rearbumper, the modified blind spot detection area can be associated withthe region of the vehicle between the passenger side mirror and about4-7 meters behind the rear bumper. Following this, in step 10508, theresponse system 188 can modify the operation of the blind spot indicatorsystem 224 so that the system warns a driver when a vehicle is furtheraway. In other words, if the system initially warns a driver if theapproaching vehicle is within 5 meters of the motor vehicle 100, or theblind spot, the system can be modified to warn the driver when theapproaching vehicle is within 10 meters of the motor vehicle 100, or theblind spot of the motor vehicle 100. Of course, it will be understoodthat in some cases, step 10506, or step 10508 can be optional steps. Inaddition, other sizes and locations of the blind spot zone are possible.

FIG. 106 illustrates an embodiment of a process of operating a blindspot indicator system in response to driver state as a function of thedriver state index of the driver. In step 10602, the response system 188receives object information. This information can include informationfrom one or more sensors capable of detecting the location of variousobjects (including other vehicles) within the vicinity of the vehicle.In some cases, for example, the response system 188 receives informationfrom a remote sensing device (such as a camera, lidar or radar) fordetecting the presence of one or more objects.

In step 10604, the response system 188 can determine the location and/orbearing of a tracked object. In step 10606, the response system 188 setsa zone threshold. The zone threshold can be a location threshold fordetermining when an object has entered into a blind spot monitoringzone. In some cases, the zone threshold can be determined using thedriver state index of the driver as well as information about thetracked object.

In step 10608, the response system 188 determines if the tracked objectcrosses the zone threshold. If not, the response system 188 proceeds tostep 10602. Otherwise, the response system 188 proceeds to step 10610.In step 10610, the response system 188 determines if the relative speedof the object is in a predetermined range. If the relative speed of theobject is in the predetermined range, it is likely to stay in the blindspot monitoring zone for a long time and can pose a very high threat.The response system 188 can ignore objects with a relative speed outsidethe predetermined range, since the object is not likely to stay in theblind spot monitoring zone for very long. If the relative speed is notin the predetermined range, the response system 188 proceeds back tostep 10602. Otherwise, the response system 188 proceeds to step 10612.

In step 106012, the response system 188 determines a warning type usingthe driver state index. In step 10614, the response system 188 sets thewarning intensity and frequency using the driver state index. Lookuptable 10618 is an example of a relationship between driver state indexand a coefficient for warning intensity. Finally, in step 10620, theresponse system 188 activates the blind spot indicator warning to alertthe driver of the presence of the object in the blind spot.

FIG. 107 illustrates an embodiment of a process for determining a zonethreshold. In step 10702, the response system 188 retrieves trackedobject information. In step 10704, the response system 188 can determinean initial threshold setting. In step 10706, the response system 188 candetermine the driver state index of the driver. In step 10708, theresponse system 188 can determine a blind spot zone coefficient. Forexample, a look-up table 10710 includes a predetermined relationshipbetween driver state index and the blind spot zone coefficient. Theblind spot zone coefficient can range between 0% and 25% in some casesand can generally increase with the driver state index. Finally, in step10712, the response system 188 can determine the zone threshold.

Generally, the zone threshold can be determined using the initialthreshold setting (determined in step 10704) and the blind spot zonecoefficient. For example, if the blind spot zone coefficient has a valueof 25%, the zone threshold can be up to 25% larger than the initialthreshold setting. In other cases, the zone threshold can be up to 25%smaller than the initial threshold setting. In other words, the zonethreshold can be increased or decreased from the initial thresholdsetting in proportion to the value of the blind spot zone coefficient.Moreover, as the value of the zone threshold changes, the size of theblind spot zone or blind spot detection area can change. For example, insome cases, as the value of the zone threshold increases, the length ofthe blind spot detection area is increased, resulting in a largerdetection area and higher system sensitivity. Likewise, in some cases,as the value of the zone threshold decreases, the length of the blindspot detection area is decreased, resulting in a smaller detection areaand lower system sensitivity.

FIG. 108 illustrates an example of an embodiment of various warningsettings according to the driver state index in the form of a lookuptable 10802. For example, when the driver's driver state index is 1, thewarning type can be set to indicator only. In other words, when thedriver is not drowsy, the warning type can be set to light-up one ormore warning indicators only. When the driver state index is 2, bothindicators and sounds can be used. When the driver's driver state indexis 3, indicators and haptic feedback can be used. For example, adashboard light can flash and the driver's seat or the steering wheelcan vibrate. When the driver's driver state index is 4, indicators,sounds and haptic feedback can all be used. In other words, as thedriver becomes more drowsy (increased driver state index), a greatervariety of warning types can be used simultaneously. It will beunderstood that the present embodiment only illustrates exemplarywarning types for different driver state indexes and in otherembodiments, any other configuration of warning types for driver stateindexes can be used.

FIGS. 109 through 116 illustrate exemplary embodiments of the operationof a collision mitigation braking system (CMBS) in response to driverstate. In some cases, a collision mitigation braking system could beused in combination with a forward collision warning system. Inparticular, in some cases, a collision mitigation braking system couldgenerate forward collision warnings in combination with, or instead of,a forward collision warning system. Moreover, the collision mitigationbraking system could be configured to further actuate various systems,including braking systems and electronic seat belt pretensioningsystems, in order to help avoid a collision. In other cases, however, acollision mitigation braking system and a forward collision warningsystem could be operated as independent systems. In the exemplarysituations discussed below, a collision mitigation braking system iscapable of warning a driver of a potential forward collision. However,in other cases, a forward collision warning could be provided by aseparate forward collision warning system.

As seen in FIG. 109, the motor vehicle 100 is driving behind targetvehicle 10902. In this situation, the motor vehicle 100 is traveling atapproximately 60 mph, while a target vehicle 10902 is slowing toapproximately 30 mph. At this point, the motor vehicle 100 and thetarget vehicle 10902 are separated by a distance D1. Because the driveris alert, however, the CMBS 220 determines that the distance D1 is notsmall enough to require a forward collision warning. In contrast, whenthe driver is drowsy, as seen in FIG. 110, the response system 188 canmodify the operation of the CMBS 220 so that a warning 11002 isgenerated during a first warning stage of the CMBS 220. In other words,the CMBS 220 becomes more sensitive when the driver is drowsy. Moreover,as discussed below, the level of sensitivity can vary in proportion tothe degree of drowsiness (indicated by the driver state index).

Referring now to FIG. 111, the motor vehicle 100 continues to approachthe target vehicle 10902. At this point, the motor vehicle 100 and thetarget vehicle 10902 are separated by a distance D2. This distance isbelow the threshold for activating a forward collision warning 11102. Insome cases, the warning could be provided as a visual alert and/or anaudible alert. However, because the driver is alert, the distance D2 isnot determined to be small enough to activate additional collisionmitigation provisions, such as automatic braking and/or automatic seatbelt pretensioning. In contrast, when the driver is drowsy, as seen inFIG. 112, the response system 188 can modify the operation of the CMBS220 so that in addition to providing the forward collision warning11102, the CMBS 220 can also automatically pretension a seat belt 11202.Also, in some cases, the CMBS 220 can apply light braking 11204 to slowthe motor vehicle 100. In other cases, however, no braking can beapplied at this point.

For purposes of illustration, the distance between vehicles is used asthe threshold for determining if the response system 188 should issue awarning and/or apply other types of intervention. However, it will beunderstood that in some cases, the time to collision between vehiclescan be used as the threshold for determining what actions the responsesystem 188 can perform. In some cases, for example, using informationabout the velocities of the host and target vehicles as well as therelative distance between the vehicles can be used to estimate a time tocollision. The response system 188 can determine if warnings and/orother operations should be performed according to the estimated time tocollision.

FIG. 113 illustrates an embodiment of a process for operating acollision mitigation braking system in response to driver state. In step11302, the response system 188 can receive target vehicle informationand host vehicle information. For example, in some cases the responsesystem 188 can receive the speed, location, and/or bearing of the targetvehicle as well as the host vehicle. In step 11304, the response system188 can determine the location of an object in the sensing area, such asa target vehicle. In step 11306, the response system 188 can determinethe time to collision with the target vehicle.

In step 11308, the response system 188 can set a first time to collisionthreshold and a second time to collision threshold. In some cases, thefirst time to collision threshold can be greater than the second time tocollision threshold. However, in other cases, the first time tocollision threshold can be less than or equal to the second time tocollision threshold. Details for determining the first time to collisionthreshold and the second time to collision threshold are discussed belowand shown in FIG. 114.

In step 11310, the response system 188 can determine if the time tocollision is less than the first time to collision threshold. If not,the response system 188 returns to step 11302. In some cases, the firsttime to collision threshold can a value above which there is noimmediate threat of a collision. If the time to collision is less thanthe first time to collision threshold, the response system 188 proceedsto step 11312.

At step 11312, the response system 188 can determine if the time tocollision is less than the second time to collision threshold. If not,the response system 188 enters a first warning stage at step 11314. Theresponse system 188 can then proceed through further steps discussedbelow and shown in FIG. 115. If the time to collision is greater thanthe second time to collision threshold, the response system 188 canenter a second warning stage at step 11316. The response system 188 canthen proceed through further steps discussed below and shown in FIG.116.

FIG. 114 illustrates an embodiment of a process for setting a first timeto collision threshold and a second time to collision threshold. In step11402, the response system 188 can determine a minimum reaction time foravoiding a collision. In step 11404, the response system 188 can receivetarget and host vehicle information such as location, relative speeds,absolute speeds, as well as any other information. In step 11406, theresponse system 188 can determine a first initial threshold setting anda second initial threshold setting. In some cases, the first initialthreshold setting corresponds to the threshold setting for warning adriver. In some cases, the second initial threshold setting correspondsto the threshold setting for warning a driver and also operating brakingand/or seat belt pretensioning. In some cases, these initial thresholdsettings can function as default setting that can be used with a driveris fully alert. Next, in step 11408, the response system 188 candetermine the driver state index of the driver.

In step 11410, the response system 188 can determine a time to collisioncoefficient. In some cases, the time to collision coefficient can bedetermined using look-up table 11412, which relates the time tocollision coefficient to the driver state index of the driver. In somecases, the time to collision coefficient increases from 0% to 25% as thedriver state index increases. In step 11414, the response system 188 canset the first time to collision threshold and the second time tocollision threshold. Although a single time to collision coefficient isused in this embodiment, the first time to collision threshold and thesecond time to collision threshold can differ according to the firstinitial threshold setting and the second initial threshold setting,respectively. Using this configuration, in some cases, the first time tocollision threshold and the second time to collision threshold can bedecreased as the driver state index of a driver increases. This allowsthe response system 188 to provide earlier warnings of potential hazardswhen a driver is drowsy. Moreover, the timing of the warnings varies inproportion to the driver state index.

FIG. 115 illustrates an embodiment of a process for operating a motorvehicle in a first warning stage of the CMBS 220. In step 11502, theresponse system 188 can select visual and/or audible warnings foralerting a driver of a potential forward collision. In some cases, awarning light can be used. In other cases, an audible noise, such as abeep, could be used. In still other cases, both a warning light and abeep could be used.

In step 11504, the response system 188 can set the warning frequency andintensity. This can be determined using the driver state index in somecases. In particular, as the driver state increases due to the increaseddrowsiness of the driver, the warning state frequency and intensity canbe increased. For example, in some cases a look-up table 11506 can beused to determine the warning frequency and intensity. In particular, insome cases as the warning intensity coefficient increases (as a functionof driver state index), the intensity of any warning can be increased byup to 25%. In step 11508, the response system 188 can apply a warningfor forward collision awareness. In some cases, the intensity of thewarning can be increased for situations where the warning intensitycoefficient is large. For example, for a low warning intensitycoefficient (0%) the warning intensity can be set to a predeterminedlevel. For higher warning intensity coefficients (greater than 0%), thewarning intensity can be increased beyond the predetermined level. Insome cases, the luminosity of visual indicators can be increased. Inother cases, the volume of audible warnings can be increased. In stillother cases, the pattern of illuminating a visual indicator or making anaudible warning could be varied.

FIG. 116 illustrates an embodiment of process of operating a motorvehicle in a second stage of the CMBS 220. In some cases, during step11602, the CMBS 220 can use visual and/or audible warnings to alert adriver of a potential collision. In some cases, the level and/orintensity of the warnings could be set according to the driver stateindex, as discussed above and shown in step 11504 of FIG. 115. Next, instep 11604, the response system 188 can use a haptic warning. Insituations where visual and/or audible warnings are also used, thehaptic warning can be provided simultaneously with the visual and/oraudible warnings. In step 11606, the response system 188 can set thewarning frequency and intensity of the haptic warning. This can beachieved using look-up table 11608, for example. Next, in step 11610,the response systems 188 can automatically pretension a seat belt inorder to warn the driver. The frequency and intensity of the tensioningcan vary as determined in step 11606. In step 11612, the response system188 can apply light braking automatically in order to slow the vehicle.In some cases, step 11612 can be optional step.

FIG. 117 illustrates an embodiment of a process of operating anavigation system in response to driver state. In some embodiments, someof the following steps could be accomplished by a response system 188 ofa motor vehicle. In some cases, some of the following steps can beaccomplished by an ECU 106 of a motor vehicle. In other embodiments,some of the following steps could be accomplished by other components ofa motor vehicle, such as vehicle systems 126. In still otherembodiments, some of the following steps could be accomplished by anycombination of systems or components of the vehicle. It will beunderstood that in some embodiments one or more of the following stepscan be optional. For purposes of reference, the following methoddiscusses components shown in FIGS. 1A, 1B through 3, including theresponse system 188.

In step 11702, the response system 188 can receive drowsinessinformation. In step 11704, the response system 188 can determine if thedriver is drowsy. If the driver is not drowsy, the response system 188proceeds back to step 11702. Otherwise, the response system 188 proceedsto step 11706. In step 11706, the response system 188 can turn offnavigation system 230. This can help reduce driver distraction.

FIG. 118 illustrates an embodiment of a process of operating a failuredetection system in response to a driver state. In some embodiments,some of the following steps could be accomplished by a response system188 of a motor vehicle. In some cases, some of the following steps canbe accomplished by an ECU 106 of a motor vehicle. In other embodiments,some of the following steps could be accomplished by other components ofa motor vehicle, such as the failure detection system 244 and/or thevehicle systems 126. In still other embodiments, some of the followingsteps could be accomplished by any combination of systems or componentsof the vehicle. It will be understood that in some embodiments one ormore of the following steps can be optional. For purposes of reference,the following method discusses components shown in FIGS. 1A, 1B through3, including the response system 188.

At step 11802, the method includes receiving drowsiness information. Insome cases, the drowsiness information includes whether a driver is in anormal state or a drowsy state. Moreover, in some cases, the drowsinessinformation could include a value indicating the level of drowsiness,for example on a scale of 1 to 10, with 1 being the least drowsy and 10being the drowsiest. In some embodiments, other types of information canbe received at step 11802, for example, physiological monitoringinformation, behavioral monitoring information, vehicular monitoringinformation, and other monitoring information from the vehicle systems126 and the monitoring systems 300.

At step 11804, the method includes determining if the driver is drowsybased on the drowsiness information. If the driver is not drowsy, theresponse system 188 returns back to step 11802. If the driver is drowsy,the response system 188 proceeds to step 11806.

In step 11806, the method includes receiving vehicle information. Insome cases, the ECU 106 and/or the response system 188 can receive thevehicle information from one or more vehicle systems 126. In othercases, the vehicle information can be received directly from the one ormore vehicle systems 126. In some embodiments, a vehicular state can bedetermined at step 11806 based on the vehicle information.

In step 11808, the method includes modifying one or more failurethresholds of the failure detection system based on the drowsinessinformation and the vehicle information. It is appreciated, that in someembodiments, the failure thresholds can be modified based on thedrowsiness information only and that step 11806 can be omitted. Theresponse system 188 can modify one or more failure thresholds of thefailure detection system 244 for one or more vehicle systems 126. It isunderstood that the response system 188 can modify one or more failurethresholds specific to a vehicle system (e.g., the failure threshold fora braking system can be different from the failure threshold for anelectric power steering system). Modifying the failure threshold changesthe sensitivity of the detection of failure in the corresponding vehiclesystem. For example, in a situation where the driver is drowsy, thesensitivity of the detection of failure in the corresponding vehiclesystem can be increased. In one embodiment, the threshold is modified asa function of the driver state and/or vehicular state.

In one embodiment, at step 11808, modifying the failure threshold isbased on a function of the driver state. For example, the failurethreshold can be decreased as the driver state index increases (e.g.,indicating drowsiness). The failure detection system 244 may include alookup table 11810. The lookup table 11810 shows example control typesof the failure thresholds according to the driver state index. Forexample, when the driver state index is 1 or 2, the control type can beset to “no change.” In these situations, the response system 188 may notmodify the failure threshold. When the driver state index of the driveris 3, which can indicate that the driver is somewhat drowsy, theresponse system 188 can set the control type to “moderate change.” Inthis situation, the response system 188 may modify the failure thresholdslightly, for example, the failure threshold can be decreased slightly(e.g., therefore increasing the failure sensitivity slightly). When thedriver state index of the driver is 4, which can indicate that thedriver is drowsy, the response system 188 can set the control type to“significant change” (e.g., considerable change). In this situation, theresponse system 188 may modify the failure threshold greatly, forexample, the failure threshold can be decreased greatly (e.g., thereforeincreasing the failure sensitivity greatly).

Referring now to FIG. 119, a diagram showing exemplary failure detectionby a failure detection system is shown. FIG. 119 will be described withrespect to detecting a failure in a control signal 11902 of anelectronic power steering system, however, it is appreciated thatfailure detection can apply to any vehicle system. In FIG. 119,exemplary failure thresholds 11904 and 11906 indicate failure thresholdswhere the failure detection system 244 executes a fail-safe function(e.g., system shutdown). Exemplary control thresholds 11908 and 11910indicate thresholds where the failure detection system 244 executes anon-fail-safe function (e.g., controlling a vehicle system).

In FIG. 119, the failure detection system 244 receives the controlsignal 11902 over a period of time from, for example, an electronicpower steering system 132. As an illustrative example, the controlsignal 11902 can be a signal indicating a steering angle (e.g.,corresponding to a rotation angle of the steering wheel). In anotherexample, the control signal 11902 can be a signal from another type ofsteering wheel sensor. The failure detection system 244 monitors thecontrol signal 11902 and compares the control signal 11902 to thethresholds. At points 11912, 11914 and 11916, the control signal 11902meets the control threshold 11908. At these points, the failuredetection system 244 executes a no fail-state function to help controland/or mitigate system shut down. For example, the failure detectionsystem 244 may control a braking system to apply braking when thecontrol signal 11902 meets a control threshold.

At point 11918, the control signal 11902 meets the failure threshold11904. Accordingly, the failure detection system 244 executes afail-safe function and shuts down the electronic power steering system132 indicating a system failure has occurred. According to the methodsand systems described herein (e.g., FIG. 118), the failure threshold11904 can be modified based on the driver state and/or the situation inwhich the motor vehicle and/or vehicle system is operating. As shown inFIG. 119, an exemplary modified failure threshold 11920 is shown.Accordingly, at point 11922, the control signal 11902 meets the modifiedfailure threshold 11920. This causes the failure detection system 244 toexecute a fail-safe function and shut down the electronic power steeringsystem 132 at a time t (e.g., an earlier time) than the failure detectedat point 11918 based on the original failure threshold 11904.

As will be discussed in further detail herein, in addition to modifyingfailure thresholds, the failure detection system 244 can also controlone or more vehicle systems based on the driver state and an operatingcondition of the vehicle when a failure is detected. Referring now toFIG. 120, an embodiment of operating one or more vehicle systems inresponse to driver state and failure detection is illustrated. It isappreciated that the components of FIGS. 118 and 120 can be integratedand or organized into different processes for different embodiments.

In some embodiments, some of the following steps could be accomplishedby a response system 188 of a motor vehicle. In some cases, some of thefollowing steps can be accomplished by an ECU 106 of a motor vehicle. Inother embodiments, some of the following steps could be accomplished byother components of a motor vehicle, such as the failure detectionsystem 244 and/or the vehicle systems 126. In still other embodiments,some of the following steps could be accomplished by any combination ofsystems or components of the vehicle. It will be understood that in someembodiments one or more of the following steps can be optional. Forpurposes of reference, the following method discusses components shownin FIGS. 1A, 1B through 3, including the response system 188.

At step 12002, the method includes receiving monitoring information. Themonitoring information can include drowsiness information indicatingwhether a driver is in a normal state or a drowsy state. Moreover, insome cases, the drowsiness information could include a value indicatingthe level of drowsiness, for example on a scale of 1 to 10, with 1 beingthe least drowsy and 10 being the drowsiest. The monitoring informationcan also include other types of information, for example, physiologicalmonitoring information, behavioral monitoring information, vehicleinformation, and other monitoring information from the vehicle systems126 and the monitoring systems 300. Further, the monitoring informationcan include information from the failure detection system 244.

At step 12004, the method includes determining if a failure is detectedfor one or more vehicle systems. For example, the response system 188can receive failure information (e.g., monitoring information receivedat step 12002) about one or more vehicle systems from the failuredetection system 244. Referring to FIG. 119, a failure is detected, forexample, at failure thresholds 11904, 11906, or 11920. In anotherembodiment, the response system 188 can receive vehicle informationdirectly from vehicle systems 126 and analyze the vehicle informationbased on the thresholds of the failure detection system 244. Forexample, the response system 188 can receive a control signal 11902 froma steering system and analyze the control signal 11902 with respect tothe failure thresholds 11904, 11906, or 11920. Referring back to FIG.120, if a failure is not detected, the method returns to step 12002. Ifa failure is detected, at step 12006 it is determined if the driver isdrowsy, for example, based on the monitoring information.

If the driver is not drowsy, the method returns to step 12002. If thedriver is drowsy, at step 12008, the method includes determining avehicular state. The vehicular state can include information related tothe motor vehicle 100 of FIG. 1A and/or the vehicle systems 126,including those vehicle systems listed in FIG. 2. In some cases, thevehicle information can also be related to a driver of the motor vehicle100. Specifically, vehicle information can include vehicle conditions,vehicle behaviors, and information about the external environment of thevehicle. In some embodiments, at step 12008, vehicular information canbe received from one or more vehicle systems to determine a vehicularstate. In other embodiments, the vehicle information can be received atstep 12002. In some embodiments, at step 12008, the method can includedetermining a current vehicle operating condition. In other embodiments,at step 12008, the method can include determining a current vehiclesituation. In further embodiments, at step 12008, the method can includedetermining a hazard and/or risk level of the vehicle operatingcondition.

At step 12010, the method includes modifying one or more vehicle systemsbased on the driver state and the vehicular state. Accordingly, thevehicle systems can be adjusted to mitigate the vehicle system failureand/or mitigate the consequences of the vehicle system failure. Thevehicle systems are modified not only based on the driver state, butalso the current operating conditions and/or current situation of thevehicle. It is appreciated that in some embodiments, the vehicle systemscan be modified according to the driver state and/or the vehicular stateas described in the lookup table 11810 of FIG. 118. Further, in someembodiments, the vehicle systems can be modified according to theseverity of the failure detected.

FIG. 121 illustrates another embodiment of operating one or more vehiclesystems and modifying failure thresholds in response to driver state andfailure detection. At step 12102, the method includes receivingmonitoring information. The monitoring information can includedrowsiness information indicating whether a driver is in a normal stateor a drowsy state. Moreover, in some cases, the drowsiness informationcould include a value indicating the level of drowsiness, for example ona scale of 1 to 10, with 1 being the least drowsy and 10 being thedrowsiest. The monitoring information can also include other types ofinformation, for example, physiological monitoring information,behavioral monitoring information, vehicle information, and othermonitoring information from the vehicle systems 126 and the monitoringsystems 300. Further, the monitoring information can include informationfrom the failure detection system 244.

At step 12104 it is determined if the driver is drowsy, for example,based on the monitoring information. If the driver is not drowsy, themethod returns to step 12102. If the driver is drowsy, at step 12106,the method may include modifying one or more failure thresholds of thefailure detection system 244 based on the monitoring information anddrowsiness information. Modifying the failure threshold changes thesensitivity of the detection of failure in the corresponding vehiclesystem. For example, in a situation where the driver is drowsy, thesensitivity of the detection of failure in the corresponding vehiclesystem can be increased. In one embodiment, the threshold is modified asa function of the driver state.

At step 12108, the method includes determining if a failure is detectedfor one or more vehicle systems. For example, the response system 188can receive failure information (e.g., monitoring information receivedat step 12102) about one or more vehicle systems from the failuredetection system 244. Referring to FIG. 119, a failure is detected, forexample, at failure thresholds 11904, 11906, or 11920. In anotherembodiment, the response system 188 can receive vehicle informationdirectly from vehicle systems 126 and analyze the vehicle informationbased on the thresholds of the failure detection system 244. Forexample, the response system 188 can receive a control signal 11902 froma steering system and analyze the control signal 11902 with respect tothe failure thresholds 11904, 11906, or 11920. Referring back to FIG.121, in another embodiment, the response system 188 can compareinformation from one or more vehicle systems to determine if a failureis detected as described in U.S. application Ser. No. 14/733,836 filedon Jun. 8, 2015 and incorporated herein by reference. It is understoodthat other methods for determining and/or detecting a failure can beimplemented herein.

If a failure is not detected, the method returns to step 12102. If afailure is detected, at step 12110, the method includes determining avehicular state. The vehicular state can include information related tothe motor vehicle 100 of FIG. 1A and/or the vehicle systems 126,including those vehicle systems listed in FIG. 2. In some cases, thevehicle information can also be related to a driver of the motor vehicle100. Specifically, vehicle information can include vehicle conditions,vehicle behaviors, and information about the external environment of thevehicle. In some embodiments, at step 12110, vehicular can be receivedfrom one or more vehicle systems to determine the vehicular state. Inother embodiments, the vehicle information can be received at step12102. In some embodiments, at step 12110, the method can includedetermining a current vehicle operating condition. In other embodiments,at step 12110, the method can include determining a current vehiclesituation. In further embodiments, at step 12110, the method can includedetermining a hazard and/or risk level of the vehicle operatingcondition.

At step 12112, the method includes modifying one or more vehicle systemsbased on the driver state and the vehicular state. Accordingly, thevehicle systems can be adjusted to mitigate the vehicle system failureand/or mitigate the consequences of the vehicle system failure. Thevehicle systems are modified not only based on the driver state, butalso the current operating conditions and/or current situation of thevehicle. It is appreciated that in some embodiments, the vehicle systemscan be modified according to the driver state and/or the vehicular stateas described in the lookup table 11810 of FIG. 118. Further, in someembodiments, the vehicle systems can be modified according to theseverity of the failure detected.

Specific examples of modifying one or more vehicle systems according tothe process of FIGS. 118, 120 and/or 121 will now be discussed. It isunderstood that the follow examples are illustrative in nature and thatother vehicle systems can be modified. Referring again to FIG. 120, at12004 it is determined based on monitoring information from the failuredetection system 244 and/or the engine 104 that a vehicle transmissionsystem is in a failure state. For example, as shown in FIG. 122A, theeffect of a vehicle transmission system in a failure state is shown.Here, the motor vehicle 100 is travelling on a road 12202 (e.g., a hill)and the vehicle transmission system (not shown) of the motor vehicle 100is detected as being in a fail state (e.g., the motor vehicle 100 isrolling back on the road 12202).

Accordingly, at step 12006, it is determined if the driver is drowsy. IfYES, at step 12008 a vehicular state is determined. In this example, thevehicular state is determined based on vehicle information about thevehicle and the environment of the vehicle (e.g., current operatingparameters and/or a current situation). For example, in FIG. 122A, themotor vehicle 100 is on a road (e.g., a hill, a road with a steep gradeincline) 12202. Other information can include weather conditions (e.g.,icy road) and/or roll back speed. Based on at least one of the driverstate and the vehicular state, one or more vehicle systems at step 12010are modified. For example, the electric parking brake system 210 can beapplied. In another embodiment, other modifications to other brakingsystems can be applied, for example, a brake assist system 206, anautomatic brake prefill system 208, among others, can be modified.

In another example, shown in FIG. 122B, a vehicular state can includeinformation about objects around the vehicle, detected for example, by ablind spot indicator system 224, a lane monitoring system 228, amongothers. In FIG. 122B, a target vehicle 12204 is shown behind the motorvehicle 100. Accordingly, modifying the one or more vehicle systems caninclude modifying the braking systems according to a distance 12206between the target vehicle 12204 and the motor vehicle 100. For example,if the target vehicle 12204 is very close to the motor vehicle 100, theelectric parking brake system 210 may be applied immediately.

As another illustrative example and referring again to the method ofFIG. 120, it may be determined at step 12004 from monitoring informationreceived at step 12002 that vehicle acceleration is in a failure state.For example, the vehicle may be experience sudden accelerationunexpectedly without input from the driver 102 (e.g., via an acceleratorpedal). Accordingly, at step 12006, it is determined if the driver isdrowsy. If YES, at step 12008 a vehicular state is determined. In thisexample, the vehicular state determined based on vehicle informationabout the vehicle and the environment of the vehicle (e.g., currentoperating parameters and/or a current situation). For example, as shownin FIG. 123, the motor vehicle 100 in a failure state where suddenacceleration is detected. The vehicular state can include informationabout objects around the motor vehicle 100, for example the targetvehicle 12302 in front of the motor vehicle 100 and the distance 12304between the motor vehicle 100 and the target vehicle 12302. Accordingly,at step 12010, the vehicle systems are modified based on at least one ofthe driver state and the vehicular state. For example, a brake assistsystem 206 can be actuated to begin braking the vehicle. The braking canbe based on the distance 12304 between the target vehicle 12302 and themotor vehicle 100 to avoid a collision with the target vehicle 12302. Ifthe target vehicle 12302 is not present, the brake assist system 206 maybe actuated to brake at a slower rate than if the target vehicle 12302was present.

As another illustrative example and referring again to the method ofFIG. 120, it may be determined at step 12004 from monitoring informationreceived at step 12002 that the electronic power steering system 132 isin a failure state (e.g., loss of steering, steering circuit brake).Accordingly, at step 12006, it is determined if the driver is drowsy. IfYES, at step 12008 a vehicular state is determined. In this example, thevehicular state determined based on vehicle information about thevehicle and the environment of the vehicle (e.g., current operatingparameters and/or a current situation). For example, as shown in FIG.124, the motor vehicle 100 is in a failure state where there is a suddenloss of steering. Here, a target vehicle 12402 is detected in a blindspot monitoring zone 12404 by a blind spot indicator system 224.Further, a potential lane deviation (e.g., caused by the sudden loss ofsteering) can be detected towards the center lane 12406 by a lanedeparture warning system 222. Accordingly, at step 12010, the vehiclesystems are modified based on at least one of the driver state and thevehicular state.

In this example, the steering wheel 134 can be actuated and turned in adirection away from the target vehicle 12402. In another embodiment, alane keep assist system 226 can be actuated to keep the motor vehicle100 in the current lane. In another embodiment, the response system 188can actuate an auto control status (e.g., vehicle mode selector system238) and/or a braking system to safely stop the vehicle. For example,the response system 188 can activate the automatic cruise control system216 and the lane keep assist system 226 to slow down the vehicle, keepthe vehicle in a current lane until the vehicle comes to a completestop.

It will be appreciated that the exemplary operational responsesdiscussed in Section VI can also apply to methods and systems utilizinga plurality of driver states, a combined driver state, and/or avehicular state. Thus, the driver state index discussed in the exemplaryoperational responses can be substituted with more than one driver stateand/or a combined driver state index as determined by the methods andsystems discussed in Section IV. Exemplary operational responses basedon one or more driver states (e.g., multi-modal neural network of driverstates) and/or vehicular states will now be discussed. However, it isappreciated that these examples are illustrative in nature and othercombinations of vehicle systems, monitoring systems and responses can becontemplated.

Referring now to FIG. 125, a flow chart of an illustrative process ofcontrolling vehicle systems according to combined driver state indexusing heart rate information and eye movement information according toan exemplary embodiment is shown. In step 12502, the method includesreceiving heart rate information, from for example a heart ratemonitoring system that senses heart rate using a bio-monitoring sensor180 embedded in the vehicle seat 168 (FIG. 1A). In step 12504, the firstdriver state is determined based on the heart rate information. Thus, inthis embodiment, the first driver state is a physiological driver state.In step 12506, the method includes receiving head and/or eye movementinformation, from for example, an optical sensing device 162, theeye/facial movement monitoring system 332 and/or the head movementmonitoring system 334. In step 12508, a second driver state isdetermined based on the eye movement information. Thus, in thisembodiment, the second driver state is a behavioral driver state.

In step 12510, it is determined if the first driver state meets a firstdriver state threshold. If YES, in step 12512, the first driver state isconfirmed with another driver state, namely, the second driver state. Instep 12514, it is determined if the second driver state meets a seconddriver state threshold. If YES, at step 12516, a combined driver stateindex is determined based on the first driver state and the seconddriver state. In step 12518, control of one or more vehicle systems ismodified based on the combined driver state index. For example, anantilock brake system 204 can be modified based on the combined driverstate index similar to the methods and systems described in the FIGS. 76and 77. It is understood that the steps of FIG. 125 can be reorganizedfor different embodiments. For example, as discussed in Section IV, thecombined driver state index can be determined with or without thresholdsand/or with or without confirmation with another driver state. Further,the thresholds can be implemented at different points in the process ofFIG. 125, for example, after confirmation. It is also appreciated thatthe process of FIG. 125 can include more than two driver states and/or avehicular state.

FIG. 126 illustrates a flow chart of an illustrative process ofcontrolling vehicle systems according to combined driver state indexsimilar to FIG. 125, but using heart rate information and steeringinformation. In step 12602, the method includes receiving heart rateinformation, from for example a heart rate monitoring system that sensesheart rate using a bio-monitoring sensor 180 embedded in the vehicleseat 168 (FIG. 1A). In step 12604, the first driver state is determinedbased on the heart rate information. Thus, in this embodiment, the firstdriver state is a physiological driver state. In step 12606, the methodincludes receiving steering information, from for example, theelectronic stability control system 202. In step 12608, a second driverstate is determined based on the steering information. Thus, in thisembodiment, the second driver state is a vehicular-sensed driver state,since the steering information is associated with the driver 102.

In step 12610, it is determined if the first driver state meets a firstdriver state threshold. If YES, in step 12612, the first driver state isconfirmed with another driver state, namely, the second driver state. Instep 12614, it is determined if the second driver state meets a seconddriver state threshold. If YES, at step 12616, a combined driver stateindex is determined based on the first driver state and the seconddriver state. In step 12618, control of one or more vehicle systems ismodified based on the combined driver state index. For example, a brakeassist system 206 can be modified based on the combined driver stateindex similar to the methods and systems described in the FIGS. 80 and81. It is understood that the steps of FIG. 126 can be reorganized fordifferent embodiments. For example, as discussed in Section IV, thecombined driver state index can be determined with or without thresholdsand/or with or without confirmation with another driver state. Further,the thresholds can be implemented at different points in the process ofFIG. 126, for example, after confirmation. It is also appreciated thatthe process of FIG. 126 can include more than two driver states and/or avehicular state.

FIG. 127 illustrates a flow chart of an illustrative process ofcontrolling vehicle systems according to combined driver state indexsimilar to FIGS. 125 and 126, but using head movement information andacceleration/deceleration information. In step 12702, the methodincludes receiving head movement information, from for example a headmovement monitoring system 334. In step 12704, the first driver state isdetermined based on the head movement information. As an illustrativeexample, the first driver state can indicate a number of head nods overa period of time as determined by the head movement monitoring system334. In step 12706, the method includes receiving acceleration and/ordeceleration information, from for example, the electronic stabilitycontrol system 202. In step 12708, a second driver state is determinedbased on the acceleration and/or deceleration information. As anillustrative example, the second driver state can indicate a number ofaccelerations over a period of time.

In step 12710, it is determined if the first driver state meets a firstdriver state threshold. For example, the first driver state thresholdcan be a number of head nods over a period of time indicating a drowsydriver. If YES, in step 12712, the first driver state is confirmed withanother driver state, namely, the second driver state. In step 12714, itis determined if the second driver state meets a second driver statethreshold. For example, the second driver state can be a number ofaccelerations over a period of time indicating a drowsy driver. If YES,at step 12716, a combined driver state index is determined based on thefirst driver state and the second driver state. If no, the processreturns to receiving monitoring information. In step 12718, control ofone or more vehicle systems is modified based on the combined driverstate index. It is understood that the steps of FIG. 127 can bereorganized for different embodiments. For example, as discussed inSection IV, the combined driver state index can be determined with orwithout thresholds and/or with or without confirmation with anotherdriver state. Further, the thresholds can be implemented at differentpoints in the process of FIG. 127, for example, after confirmation. Itis also appreciated that the process of FIG. 127 can include more thantwo driver states and/or a vehicular state.

An exemplary operational response based on one or more driver states anda vehicular state will now be described. FIG. 128 illustrates a flowchart of an illustrative process of controlling vehicle systemsaccording to combined driver state index and a vehicular state includingthresholds. At step 12802, the response system 188 determines a firstdriver state. In one embodiment, the first driver state is at least oneof a physiological driver state, a behavioral driver state, and avehicular-sensed driver state. As an illustrative example, the firstdriver state of FIG. 128 is a physiological driver state based on, forexample, heart rate information of the driver.

At step 12804, the response system 188 determines a second driver state.In one embodiment, the second driver state is at least one of aphysiological driver state, a behavioral driver state, and avehicular-sensed driver state. Thus, referring again to the illustrativeexample, in FIG. 128, the first driver state is a behavioral driverstate based on, for example, gesture recognition information from thedriver. It is appreciated that a third driver state can also bedetermined and utilized in the process of FIG. 128. In an embodimentwith a third driver state, in FIG. 128, the third driver state is atleast one of a physiological driver state, a behavioral driver state anda vehicular-sensed driver state.

At step 12806, the response system 188 determines a vehicular statebased on vehicle information. As an illustrative example, in FIG. 128,the vehicular state is based on a current vehicle speed. Each of thefirst driver state, the second driver state, and the vehicular state canoptionally be passed through respective thresholds (e.g., T₁, T₂, T_(v))by the response system 188. With regards to the first driver state andthe second driver state, at step 12808, the first driver state and thesecond driver state can be confirmed, as discussed herein. In oneembodiment, step 12808 can be a decision step. Thus, if the outcome ofstep 12808 is YES (i.e., driver states are confirmed), the responsesystem can proceed to step 12810 to determine a combined driver statebased on the first drive state and the second driver state.

In another embodiment, the first driver state and the second driverstate may not be confirmed, but can be used by the response system 188to determine a combined driver state index at step 12810. Further, thecombined driver state index can be confirmed and/or compared to thevehicular state by the response system 188 at step 12812. In oneembodiment, step 12812 can be a decision step. Thus, if the outcome ofstep 12812 is YES (i.e., the combined driver state is confirmed with thevehicular state), the response system 188 can modify the control of thevehicle systems at step 12814 based on the combined driver state indexand the vehicular state.

An operational illustrative example will now be described. The firstdriver state (i.e., a heart rate of the driver) meets threshold T₁indicating a normal driver state (e.g., normal heart rate for thedriver). The second driver state (i.e., gesture recognition information)meets threshold T₂ indicating a distracted driver state (e.g., thedriver is using gestures that indicate the driver is engaged in otheractivities other than the task of driving, for example, on the phone).The vehicular state (i.e., current vehicle speed) meets threshold T_(v)indicating a high risk level (e.g., the current vehicle speed is high).

In one embodiment, at step 12808, the first driver state and the seconddriver state can be confirmed. In this example, in some embodiments, ifthe first driver state is normal (i.e., 0) and the second driver stateis distracted (i.e., 1), the response system 188 can proceed to step12810 to determine a combined driver state index based on the firstdriver state and the second driver state.

At step 12812, the combined driver state index is confirmed with thevehicular state. In this embodiment, if the combined driver state indexindicates a distracted driver and the vehicular state indicates a highrisk, the response system 188 can modify the control of the vehiclesystems at step 12814. For example, the response system 188 can alertthe driver visually (e.g., visual devices 140) to their current speedand/or alert the driver about their distracted state. In anotherembodiment, the response system 188 could restrict the use of the phonethe driver is using via, for example, the navigation system 230. Inanother embodiment, the response system 188 could modify the lanedeparture warning system 222 and/or the blind spot indicator system 224to warn the driver earlier of potential collisions or to prevent thevehicle from changing lanes if the driver is distracted.

As another illustrative example, if the vehicular state is based oncurrent traffic information and meets threshold T_(v) indicating a lowrisk level (e.g., no traffic or low traffic), at step 12812 when thevehicular state is confirmed and/or compared to the combined driverstate index, the response system 188 may not restrict the use of thedriver phone, but instead only provide a visual warning to the driver.As can be understood, various combinations and modifications to the oneor more vehicle systems are possible.

B. Exemplary Operational Response of More than One Vehicle System toDriver State

In some embodiments, a vehicle can include provisions for modifyingdifferent vehicle systems in response to driver state. Further, in someembodiments, the vehicle can include provisions for modifying differentvehicle systems in response to driver state, a combined driver state,and/or a vehicular state, substantially and/or simultaneously. Themultiple vehicle systems, in some embodiments, can communicateinformation to each other for proper modification of control of one ormore vehicle systems. The number of vehicle systems that can besimultaneously activated in response to driver state is not limited. Forexample, in some cases, one or more vehicle systems can be configured tocommunicate with one another in order to coordinate responses to ahazard or other driving condition. In some cases, the hazard or otherdriving condition is a vehicular state as discussed above in Section V.In some cases, a centralized control unit, such as an ECU, can beconfigured to control various different vehicle systems in a coordinatedmanner to address hazards or other driving conditions.

For purposes of clarity, the term hazard, or hazardous condition, isused throughout this detailed description and in the claims to refergenerally to one or more objects and/or driving scenarios that pose apotential safety threat to a vehicle. For example, a target vehicletraveling in the blind spot of a driver can be considered a hazard sincethere is some risk of collision between the target vehicle and the hostvehicle should the driver turn into the lane of the target vehicle.Additionally, a target vehicle that is traveling in front of a hostvehicle can also be categorized as a hazard for purposes of operating aresponse system. Furthermore, the term hazard is not limited todescribing a target vehicle or other remote object. In some cases, forexample, the term hazard can be used to describe one or more hazardousdriving conditions that increase the likelihood of an accident. Further,as mentioned above, the term hazard or hazardous condition level canrefer to a vehicular state.

Modifying control of one or more vehicle systems based on informationfrom more than one vehicle system, the driver state, and in someembodiments, the driver state relative to the information from thevehicle systems (e.g., hazards, risks), allows for a customizedresponse. This results in a level of control appropriate for the currentsituation (e.g., hazard, risk level) and the current driver state. Forexample, in some cases when a driver is fully attentive (e.g., notdrowsy), control of some vehicle systems can be overridden orsuppressed. This gives the driver full control of the vehicle. In somecases when a driver is somewhat attentive (e.g., somewhat drowsy)control of some vehicle systems may be slightly modified. This gives thedriver some control of the vehicle. In other cases, where the driver isdistracted (e.g., drowsy), some vehicle systems may be significantlymodified. This gives the driver less control of the vehicle. Further, insome cases, when the driver is very distracted (e.g., very drowsy and/orpossibly asleep), some vehicle systems may be modified to automaticallycontrol the vehicle, in a full or semi-autonomous mode. In this case,the driver has little to no control of the vehicle and most control orfull control is passed to the vehicle.

Accordingly, the embodiments discussed herein will discuss generalprovisions for sensing driver state and modifying the operation of oneor more vehicle systems based on the driver state. More specifically,embodiments providing intra-vehicle communication and control andembodiments providing semi-autonomous and/or fully autonomous controlwill be discussed. It is understood that the embodiments discussedherein can implement any of the vehicle systems, monitoring systems, andsystems for determining driver state and/or combined driver statediscussed above. Further, it is understood that methods and systemsdiscussed herein are not limited to use with a driver. In otherembodiments, these same methods and systems could be applied to anyoccupant of a vehicle. In other words, a response system can beconfigured to detect if various other occupants of a motor vehicle aredistracted. Moreover, in some cases, one or more vehicle systems couldbe modified accordingly.

Referring now to the drawings, FIG. 129 illustrates a schematic view ofan embodiment of a response system 12900 for modifying control of one ormore vehicle systems. The response system 12900 can include variousvehicle systems that can be modified in response to driver state,including drowsy driving. The response system 12900 can be the sameand/or similar to the response system 188 of FIG. 1A. Further, in somecases, the response system 12900 can include a centralized control unit,such as an electronic control unit (ECU) 12902. The ECU 12902 can be thesame and/or similar to the ECU 106 of FIGS. 1A and 1B. Examples ofdifferent vehicle systems that can be incorporated into the responsesystem 12900 include any of the vehicle systems described above andshown in FIG. 2 as well as any other vehicle systems. It should beunderstood that the systems shown in FIG. 2 are only intended to beexemplary and in some cases, some other additional systems can beincluded. In other cases, some of the systems can be optional and notincluded in all embodiments.

In some embodiments, the response system 12900 includes the electronicpower steering system 132, the touch steering wheel system 134, thevisual devices 140, the audio devices 144, the tactile devices 148, theuser input devices 152, the infotainment system 154, the electronicstability control system 202, the antilock brake system 204, the brakeassist system 206, the automatic brake prefill system 208, the EPBsystem 210, the low speed follow system 212, the cruise control system214, the automatic cruise control system 216, the collision warningsystem 218, the collision mitigation braking system 220, the lanedeparture warning system 222, the blind spot indicator system 224, thelane keep assist system 226, the lane monitoring system 228, thenavigation system 230, the hands free portable device system 232, theclimate control system 234, the electronic pretensioning system 236, thevehicle mode selector system 238, the turn signal control system 240,the headlight control system 242, and the failure detection system 244,which are referred to collectively as the vehicle systems 126.

In other embodiments, the response system 12900 can include additionalvehicle systems. In still other embodiments, some of the systemsincluded in FIG. 129 can be optional. Moreover, in some cases, theresponse system 12900 can be further associated with various kinds ofmonitoring devices including any of the monitoring systems and devicesdiscussed above (for example, optical devices, various types of positionsensors, monitoring devices or systems, autonomic monitoring devices orsystems, as well as any other devices or systems and systems shown inFIG. 3).

The response system 12900 can also provisions for centralized controlof, and/or communication between, various vehicle systems, using, forexample, the ECU 12902. The ECU 12902 can include a microprocessor, RAM,ROM, and software all serving to monitor and supervise components of theresponse system 12900 as well as any other components of a motorvehicle. The output of various devices is sent to the ECU 12902 wherethe device signals can be stored in an electronic storage, such as RAM.Both current and electronically stored signals can be processed by acentral processing unit (CPU) in accordance with software stored in anelectronic memory, such as ROM. The ECU 12902 can include some or all ofthe components of the ECU 106 shown in FIG. 1B.

The ECU 12902 can include a number of ports that facilitate the inputand output of information and power. The term “port” as used throughoutthis detailed description and in the claims refers to any interface orshared boundary between two conductors. In some cases, ports canfacilitate the insertion and removal of conductors. Examples of thesetypes of ports include mechanical connectors. In other cases, ports areinterfaces that generally do not provide easy insertion or removal.Examples of these types of ports include soldering or electron traces oncircuit boards.

All of the following ports and provisions associated with the ECU 12902are optional. Some embodiments can include a given port or provision,while others can exclude it. The following description discloses many ofthe possible ports and provisions that can be used, however, it shouldbe kept in mind that not every port or provision must be used orincluded in a given embodiment.

In some cases, the ECU 12902 can include a port 12904, a port 12906, aport 12908, a port 12910, a port 12912, a port 12914, a port 12916, anda port 12918 for transmitting signals to and/or receiving signals fromthe electronic power steering system 132, the touch steering wheelsystem 134, the visual devices 140, the audio devices 144, the tactiledevices 148, the user input devices 152, the infotainment system 154,the electronic stability control system 202, respectively. In somecases, the ECU 12902 can include a port 12920, a port 12922, a port12924, a port 12926, a port 12928, and a port 12930 for transmittingsignals to and/or receiving signals from the antilock brake system 204,the brake assist system 206, the automatic brake prefill system 208, theEPB system 210, the low speed follow system 212, the cruise controlsystem 214, respectively.

In some cases, the ECU 12902 can include a port 12932, a port 12934, aport 12936, a port 12938, a port 12940, a port 12942, a port 12944, anda port 12946 for transmitting signals to and/or receiving signals fromthe automatic cruise control system 216, the collision warning system218, the collision mitigation braking system 220, the lane departurewarning system 222, the blind spot indicator system 224, the lane keepassist system 226, the lane monitoring system 228, the navigation system230, respectively. In some cases, the ECU 12902 can include a port12948, a port 12950, a port 12952, a port 12954, a port 12956, a port12958, and a port 12960 for transmitting signals to and/or receivingsignals from the hands free portable device system 232, the climatecontrol system 234, the electronic pretensioning system 236, the vehiclemode selector system 238, the turn signal control system 240, theheadlight control system 242, and the failure detection system 244,respectively.

In some embodiments, the ECU 12902 can be configured to control one ormore of vehicle systems 126. For example, the ECU 12902 could receiveoutput from one or more vehicle systems 126, make control decisions, andprovide instructions to one or more vehicle systems 126. In such cases,the ECU 12902 can function as a central control unit. In other cases,however, the ECU 12902 could simply act as a relay for communicationbetween two or more of vehicle systems 126. In other words, in somecases, the ECU 12902 could passively transmit messages between two ormore of vehicle systems 126 without making any control decisions.

As discussed herein, the methods and systems allow for communicationbetween vehicle systems. FIG. 130 illustrates a schematic view of anembodiment of a first vehicle system 13002 and a second vehicle system13004, which are in communication via a network 13006. Generally,network 13006 can be any kind of network known in the art. Examples ofdifferent kinds of networks include, but are not limited to local areanetworks, wide area networks, personal area networks, controller areanetworks as well as any other kinds of networks. In some cases, network13006 can be a wired network. In other cases, network 13006 can be awireless network.

For purposes of clarity, only two vehicle systems are shown connected toone another using a network. However, in other cases, any other numberof vehicle systems could be connected using one or more networks. Forexample, in some embodiments, some or all of the vehicle systems 126,shown in FIG. 129, including the response system 12900 and ECU 12902,could be connected through a network. In such a situation, each vehiclesystem of the vehicle systems 126 can function as a node within thenetwork. Moreover, using a networked configuration allows hazardinformation to be shared between each system of the vehicle systems 126.In some cases, a vehicle system can be configured to control anothervehicle system by transmitting instructions over a network. It isunderstood that the network system described in FIG. 130 can beimplemented with the systems and methods discussed herein forcommunicating information between more than one vehicle system.

Referring now to FIG. 131, an embodiment of a process for generallycontrolling one or more vehicle systems in a motor vehicle is shown. Insome embodiments, some of the following steps could be accomplished by aresponse system 12900 of a motor vehicle. In some cases, some of thefollowing steps can be accomplished by an ECU 12902 of a motor vehicle.In other embodiments, some of the following steps could be accomplishedby other components of a motor vehicle, such as vehicle systems 126. Instill other embodiments, some of the following steps could beaccomplished by any combination of systems or components of the vehicle.It will be understood that in some embodiments one or more of thefollowing steps can be optional. For purposes of reference, thefollowing method discusses components shown in FIG. 129, including theresponse system 12900.

In step 13102, the ECU 12902 can communicate with one or more of vehiclesystems 126. In some cases, the ECU 12902 can receive various kinds ofinformation from vehicle systems 126 related to driving conditions,vehicle operating conditions, target vehicle or target objectinformation, hazard information, as well as any other information. Insome cases, each system of vehicle systems 126 can transmit differentkinds of information since each system can utilize different kinds ofinformation while operating. For example, the cruise control system 214can provide the ECU 12902 with information related to a current vehiclespeed. However, the electronic power steering system 132 may not monitorvehicle speed and therefore may not transmit vehicle speed informationto the ECU 12902. In some cases, some systems may send overlappinginformation. For example, the multiple systems of vehicle systems 126may transmit information gathered from remote sensing devices.Therefore, it will be understood that information received by the ECU12902 from a particular vehicle system may or may not be unique relativeto information received from other systems of vehicle systems 126.

In some cases, the ECU 12902 can receive driver state information (suchas a level of drowsiness as characterized using a driver state index).In some cases, driver state information could be received directly fromvehicle systems 126. In other cases, driver state information could bereceived from monitoring devices or systems as discussed above. It isunderstood that communication as discussed in step 13102 can befacilitated by the communication network 13006 shown in FIG. 130 above.

Referring again to FIG. 131, in step 13104, the ECU 12902 can evaluatepotential hazards. In some cases, the potential hazard can be evaluatedas a vehicular state. In some cases, one or more vehicle systems 126 cantransmit hazard information to ECU 12902 that can characterize a giventarget vehicle, object or driving situation as a hazard. In other cases,the ECU 12902 can interpret data provided by one or more vehicle systems126 to determine if there are any potential hazards. In other words, thecharacterization of a vehicle, object, or driving situation as a hazardcan be accomplished within an individual vehicle system of vehiclesystems 126 and/or by the ECU 12902. In some cases, a target vehicle,object or driving situation can be considered a hazard by one system butnot another. For example, information about a target vehicle travelingbeside the host vehicle can be used by the blind spot indicator system224 to categorize the target vehicle as a hazard, but using the sameinformation the low speed follow system 212 may not categorize thetarget vehicle as a hazard, since the low speed follow system 212 isprimarily concerned with other vehicles located in front of the hostvehicle.

In situations where the ECU 12902 determines that a potential hazardexists, the ECU 12902 can decide to modify the control of one or morevehicle systems 126 in response to the potential hazard at step 13106.In one embodiment, where the ECU 12902 determines that a potentialhazard does not exist, the ECU 12902 can decide to modify and/or notmodify the control of one or more vehicle systems 126. In some cases,the ECU 12902 can modify the control of one vehicle system. In othercases, the ECU 12902 can modify the control of two or more vehiclesystems substantially simultaneously. In some cases, the ECU 12902 cancoordinate the modified operation of two or more vehicle systems inorder to enhance the response of a vehicle to a potential hazard. Forexample, simultaneously modifying the operation of vehicle systems thatpassively warn a driver of hazards and vehicle systems that activelychange some parameter of vehicle operation (such as speed, brakinglevels, deactivating cruise control, etc.) according to driver state canprovide a more robust response to hazards. This configuration allows theECU 12902 to provide responses that supply just the right level ofassistance depending on the state of the driver.

In some embodiments, the ECU 12902 can maintain full control over allvehicle systems 126. In other embodiments, however, some vehicle systems126 can operate independently with some input or control from the ECU12902. In such cases, the ECU 12902 can receive information from systemsthat are already in a modified control mode, and can subsequently modifythe operation of additional vehicle systems to provide a coordinatedresponse to a potential hazard. Moreover, by analyzing the response ofsome vehicle systems, ECU 12902 can override automatic control of othervehicle systems in response to a hazard. For example if a first vehiclesystem detects a hazard, but a second vehicle system does not, the ECU12902 can instruct the second vehicle system to behave as though ahazard is present. As another example if a first vehicle system detectsa hazard, but a second vehicle system does not, the ECU 12902 caninstruct the first vehicle system to behave as though a hazard is notpresent. As a further example, if a first or a second vehicle systemdetects a hazard, but the driver state indicates the driver is attentiveor knows (e.g., confirms) the hazard is present, the ECU 12902 caninstruct the first and/or second vehicle system to behave as though thehazard is not present.

In embodiments where the ECU 12902 acts in a passive manner, ECU 12902can function to receive hazard warnings from one vehicle system andtransmit the hazard warnings to one or more additional vehicle systems126. With this configuration, the ECU 12902 can distribute hazardwarnings between two or more of the vehicle systems 126 to enhance theoperation of the response system 12900.

Referring now to FIGS. 132 and 133, other embodiments of processes forcommunicating information and controlling one or more vehicle systems ina motor vehicle are illustrated. The methods described with reference toFIGS. 132 and 133 generally describe modifying one or more vehiclesystems, wherein the modifying can include modifying the control of thevehicle at different levels, for example, no control, partial control,or full control of a vehicle system. In some embodiments, some of thefollowing steps could be accomplished by a response system 12900 of themotor vehicle 100. In some cases, some of the following steps can beaccomplished by an ECU 12902 of a motor vehicle. In other embodiments,some of the following steps could be accomplished by other components ofa motor vehicle, such as vehicle systems 126. In still otherembodiments, some of the following steps could be accomplished by anycombination of systems or components of the vehicle. It will beunderstood that in some embodiments one or more of the following stepscan be optional. For purposes of reference, the following methoddiscusses components shown in FIG. 129, including the response system12900.

Referring now to FIG. 132, at step 13202, the ECU 12902 can receiveinformation from one or more of vehicle systems 126 and/or monitoringsystems 300. This information can include sensed information as well asinformation characterizing the operation of vehicle systems 126. Forexample, in some cases, the ECU 12902 could receive information fromelectronic stability control system 202 including wheel speedinformation, acceleration information, yaw rate information as well asother kinds of sensed information utilized by electronic stabilitycontrol system 202. Additionally, in some cases, the ECU 12902 couldreceive information related to the operating state of electronicstability control system 202. As an example, the ECU 12902 could receiveinformation indicating that the electronic stability control system 202is actively facilitating control of the vehicle by actuating one or morewheel brakes.

In some embodiments, the ECU 12902 can optionally receive driver stateinformation from one or more of the vehicle systems 126 and/ormonitoring systems 300 during step 13202. For example, one or more ofthe vehicle systems 126 can determine a driver state index for a driver.In some cases, multiple different systems can send the ECU 12902 adriver state index or other driver state information. In otherembodiments, the ECU 12902 can receive driver state information directlyfrom one or more monitoring systems 300 rather than receiving driverstate information from one of vehicle systems 126. In such cases, theECU 12902 can be configured to determine a driver state index accordingto the monitoring information. In still other embodiments, driver stateinformation can be received from the vehicle systems 126 as well asindependently from one or more monitoring systems 300.

In step 13204, the ECU 12902 can detect a potential hazard. In someembodiments, a hazard can be detected through information provided byone or more vehicle systems 126. In some embodiments, the hazard isreferred to as a vehicular state. As an example, the ECU 12902 canreceive information from the blind spot indicator system 224 indicatingthat a target vehicle is traveling in the blind spot of the hostvehicle. In this situation, the ECU 12902 can identify the targetvehicle as a potential hazard. As another example, the ECU 12902 couldreceive information from collision warning system 218 indicating that atarget vehicle can be traveling through an intersection approximatelysimultaneously with the host vehicle. In this situation, the ECU 12902can identify the target vehicle as a potential hazard. It will beunderstood that a target vehicle or object could be designated as apotential hazard by one or more of vehicle systems 126 or by the ECU12902. In other words, in some cases, a vehicle system determines thatan object is a potential hazard and sends this information to the ECU12902. In other cases, the ECU 12902 receives information about a targetobject from a vehicle system and determines if the object should beidentified as a potential hazard.

After identifying a potential hazard, in step 13206, the ECU 12902 candetermine a risk level for the potential hazard. In other words, in step13206, the ECU 12902 determines how much of a risk a potential hazardposes. This step allows the ECU 12902 to make control decisions aboutpotential hazards that pose the greatest risk and can reduce thelikelihood of the ECU 12902 modifying operation of one or more vehiclesystems in response to a target vehicle, object, or driving situationthat does not pose much of a risk to a vehicle. Details of a method ofdetermining a risk level for a potential hazard are discussed below andshown in FIG. 133, which provides several possible sub-steps associatedwith step 13206.

The risk level determined in step 13206 could be characterized in anymanner. In some cases, the risk level could be characterized by a rangeof numeric values (for example, 1 to 10, with 1 being the lowest riskand 10 being the highest risk). In some cases, the risk level could becharacterized as either “high risk” or “low risk.” In still other cases,the risk level could be characterized in any other manner.

In step 13208, the ECU 12902 determines if the risk level associatedwith a potential hazard is high. In some cases, the ECU 12902 determinesif the risk level is high based on a predetermined risk level. Forexample, in situations where a 1 to 10 risk level scale is used, thepredetermined risk level could be 8, so that any hazard having a risklevel at 8 or above is identified to have a high risk level. In othercases, the ECU 12902 could use any other method to determine if the risklevel identified during step 13206 is high enough to require furtheraction.

If the risk level is not high, the ECU 12902 returns to step 13202.Otherwise, the ECU 12902 proceeds to step 13210. In step 13210, the ECU12902 can select one or more of the vehicle systems 126 to be modifiedin response to a potential hazard. In some cases, the ECU 12902 couldselect a single vehicle system. In other cases, the ECU 12902 couldselect two or more vehicle systems. Moreover, as discussed in furtherdetail below, the ECU 12902 can coordinate the operation of twodifferent vehicle systems of the vehicle systems 126, so that eachsystem is modified in an appropriate manner to enhance the ability of adrowsy driver to maintain good control of a vehicle. This allows somesystems to enhance the operation and control of other systems.

In step 13212, the ECU 12902 can determine the type of modified controlfor each system selected in step 13210. In some cases, the ECU 12902 canuse the driver state index of a driver to determine the control type.For example, as seen in FIG. 132, the ECU 12902 can use the driver stateindex determined in step 13214 to select a control type. An example ofvarious control type settings according to the driver state index isshown in the form of lookup table 13216. For example, when the driverstate index is 1 or 2, the control type can be set to “no control.” Inthese situations, the ECU 12902 may not adjust the operation of any ofvehicle systems 126. When the driver state index of the driver is 3,which can indicate that the driver is somewhat drowsy, the ECU 12902 canset the control of one or more of the vehicle systems 126 to “partialcontrol.” In the partial control mode, the control of one or morevehicle systems 126 can be slightly modified to help enhancedrivability. When the driver state index of the driver is 4, which canindicate that the driver is very drowsy or even asleep, the ECU 12902can set the control of one or more of the vehicle systems 126 to “fullcontrol.” In the “full control” mode, the ECU 12902 can substantiallymodify the control of one or more of the vehicle systems 126. Using thisarrangement, a vehicle system can be configured to provide additionalassistance to a driver when the driver is very drowsy, some assistancewhen the driver is somewhat drowsy, and little to no assistance when thedriver is relatively alert (not drowsy). In step 13218, the ECU 12902can modify the control of one or more selected systems of the vehiclesystems 126. In some cases, a vehicle system can be controlled accordingto the control type determined during step 13212.

FIG. 133 illustrates one embodiment of a process for determining therisk level for a potential hazard. It will be understood that thismethod is only intended to be exemplary and in other embodiments, anyother method could be used to evaluate the risk level for a potentialhazard. In step 13302, the ECU 12902 can determine the relative distancebetween the potential hazard and the host vehicle. In some cases, theECU 12902 can determine the relative distance between the host vehicleand the hazard using a remote sensing device, including radar, lidar,cameras, as well as any other remote sensing devices. In other cases,the ECU 12902 could use GPS information for the host vehicle and thehazard to calculate a relative distance. For example, the GPS positionof the host vehicle can be received using a GPS receiver within the hostvehicle. In situations where the hazard is another vehicle, GPSinformation for the hazard could be obtained using a vehiclecommunication network or other system for receiving remote vehicleinformation.

Next, in step 13304, the ECU 12902 can determine the host vehicletrajectory relative to the hazard. In step 13306, the ECU 12902 candetermine the hazard trajectory relative to the host vehicle. In somecases, these trajectories can be estimated using remote sensing devices.In other cases, these trajectories can be estimated from real-time GPSposition information. In still other cases, any other methods fordetermining trajectories for a host vehicle and a hazard (such as aremote vehicle) could be used.

By determining the relative distances as well as relative trajectoriesof the host vehicle and hazard, the ECU 12902 can determine theprobability that the host vehicle will encounter the hazard. Inparticular, using the relative distance as well as trajectoryinformation, the ECU 12902 can estimate the probability that the hostvehicle and the hazard can eventually collide. In step 13308, the ECU12902 can determine the risk level for the hazard, which is an indicatorof the likelihood that the host vehicle will encounter the hazard. Insome cases, the ECU 12902 classifies the potential hazard as presentinga high risk or a low risk to the host vehicle.

FIG. 134 illustrates an embodiment of a process for controlling one ormore vehicle systems in response to potential hazards in situationswhere the vehicle systems can be in direct communication with oneanother, such as through a network. In some cases, certain steps of theprocess are associated with a first vehicle system 13402 and certainsteps are associated with a second vehicle system 13404. In some cases,steps associated with the first vehicle system 13402 are performed bythe first vehicle system 13402 and steps associated with the secondvehicle system 13404 are performed by the second vehicle system 13404.However, in other cases, some steps associated with the first vehiclesystem 13402 can be performed by the second vehicle system 13404 or someother resource. Likewise, in other cases, some steps associated withsecond vehicle system 13404 can be performed by the first vehicle system13402 or some other resource. In still other embodiments, some of thefollowing steps could be accomplished by any combination of systems orcomponents of the vehicle. It will be understood that in someembodiments one or more of the following steps can be optional.

In step 13406, the first vehicle system 13402 can receive operatinginformation. This information can include any kind of informationincluding sensed information as well as information characterizing theoperation of vehicle systems 126. In one embodiment, the first vehiclesystem 13402 receives operating information required for the normaloperation of the first vehicle system 13402. For example, in anembodiment where the first vehicle system 13402 is a blind spotindicator system 224, the first vehicle system 13402 could receiveinformation from a camera monitoring the blind spot region beside thevehicle, information about any tracked objects within or near the blindspot region, current vehicle speed, as well as any other informationused to operate blind spot indicator system 224.

In step 13408, the first vehicle system 13402 can determine the driverstate index of a driver. This information could be determined accordingto various monitoring information received from one or monitoringdevices, such as cameras, position sensors (such as head positionsensors) autonomic monitoring systems or any other devices. In somecases, the driver state index could also be determined using informationfrom a vehicle system. For example, a system could determine that adriver is drowsy by monitoring outputs from a lane departure warningsystem 222, as previously discussed.

In step 13410, the first vehicle system 13402 can detect a potentialhazard. In some cases, the hazard is referred to as a vehicular state.In some embodiments, a hazard can be detected through informationprovided to the first vehicle system 13402. For example, in the casewhere the first vehicle system 13402 is an automatic cruise controlsystem, the first vehicle system 13402 can be configured to receiveheadway distance information through a camera, lidar, radar or otherremote sensing device. In such cases, the first vehicle system 13402 candetect remote objects, such as a vehicle, using similar remote sensingtechniques. In other cases, a hazard can be detected through informationprovided by any other vehicle system.

After identifying a potential hazard, in step 13412, the first vehiclesystem 13402 can determine a risk level for the potential hazard. Inother words, in step 13412, the first vehicle system 13402 determineshow much of a risk a potential hazard poses. This step allows the firstvehicle system 13402 to make control decisions about potential hazardsthat pose the greatest risk and can reduce the likelihood that theoperation of the first vehicle system 13402 will be modified in responseto a target vehicle, object, or driving situation that does not posemuch of a risk to a vehicle. Details of a method of determining a risklevel for a potential hazard have been discussed previously.

In step 13414, the first vehicle system 13402 determines if the risklevel associated with a potential hazard is high. In some cases, thefirst vehicle system 13402 determines if the risk level is high based ona predetermined risk level. For example, in situations where a 1 to 10risk level scale is used, the predetermined risk level could be 8, sothat any hazard having a risk level at 8 or above is identified to havea high risk level. In other cases, the first vehicle system 13402 coulduse any other method to determine if the risk level identified duringstep 13412 is high enough to require further action.

If the risk level is high, the first vehicle system 13402 proceeds tostep 13416. Otherwise, the first vehicle system 13402 returns to step13406. In step 13416, the control of the first vehicle system 13402 canbe modified according to the current driver state index. In step 13418,the first vehicle system 13402 determines if the second vehicle system13404 should be informed of the potential hazard detected by the firstvehicle system 13402. In some cases, the second vehicle system 13404 canbe informed of any hazards encountered by the first vehicle system13402. In other cases, however, one or more criteria could be used todetermine if the second vehicle system 13404 should be notified of apotential hazard detected by the first vehicle system 13402. Inembodiments where multiple vehicle systems are in communication with oneanother, a vehicle system detecting a hazard could send informationwarning all the other vehicle systems of the hazard.

In step 13420, the first vehicle system 13402 checks to see if thesecond vehicle system 13404 should be informed of the potential hazard.If second vehicle system should not be informed, the first vehiclesystem 13402 returns to step 13406. Otherwise, the first vehicle system13402 proceeds to step 13422 where information is submitted to thesecond vehicle system 13404. In some cases, the submitted informationincludes a warning and/or instructions for the second vehicle system13404 to check for a potential hazard.

In step 13424, the second vehicle system 13404 receives information fromthe first vehicle system 13402. This information can include informationrelated to the potential hazard as well as any other information. Insome instances, the information can include instructions or a requestfor the second vehicle system 13404 to check for any potential hazards.In some cases, the information can include operating information relatedto the first vehicle system 13402. Next, in step 13426, the secondvehicle system 13404 can retrieve operating information. This operatinginformation could include any type of information used during theoperation of the second vehicle system 13404, as well as operatinginformation from any other system or device of the motor vehicle.

In step 13428, the second vehicle system 13404 can check for potentialhazards as advised or instructed by the first vehicle system 13402.Then, in step 13430, the second vehicle system 13404 can determine therisk level for the potential hazard using methods similar to those usedby the first vehicle system 13402 during step 13412. In step 13432, thesecond vehicle system 13404 can determine if the risk level is high. Ifnot, the second vehicle system 13404 returns to step 13426. Otherwise,the second vehicle system 13404 proceeds to step 13434.

In step 13434, the driver state index of the driver can be determined.This can be determined using any of the methods described above.Moreover, in some cases, the driver state index can be retrieveddirectly from the first vehicle system 13402. In step 13436, the controlof second vehicle system 13404 is modified according to the driver stateindex. This method can facilitate better system response to a hazard bycoordinating the operation of multiple vehicle systems and modifying theoperation of each system according to the driver state index.

As discussed above, the processes for controlling one or more vehiclesystems can include communication (e.g., intra-vehicle communication)between various vehicle systems. The vehicle systems can independentlycollect information, determine hazards, determine risk levels, determinedriver states, modify control of vehicle systems, and share thisinformation with other vehicle systems. This allows the vehicle systemsto work in coordination with one another. FIG. 135A illustrates anotherembodiment of a process for controlling one or more vehicle systems in amotor vehicle including a first vehicle system 13502 and a secondvehicle system 13504. In some embodiments, some of the following stepscould be accomplished by a response system 12900 of the motor vehicle100. In some cases, some of the following steps can be accomplished byan ECU 12902 of a motor vehicle. In other embodiments, some of thefollowing steps could be accomplished by other components of a motorvehicle, such as vehicle systems 126. In still other embodiments, someof the following steps could be accomplished by any combination ofsystems or components of the vehicle. It will be understood that in someembodiments one or more of the following steps can be optional.

At step 13508, the method includes receiving information from a firstvehicle system 13502. In some embodiments, the method can also includereceiving information from monitoring systems and/or other variousvehicle systems (e.g., physiological information, behavioralinformation, vehicle information). At step 13510, the method includesdetecting a potential hazard based on the information from step 13508.At step 13512, the method includes detecting a risk level associatedwith the potential hazard.

At step 13514, the method includes determining a driver state, forexample, based on information from the first vehicle system 13502 and/orthe second vehicle system 13504. For example, information can bereceived from the second vehicle system 13504 at step 13522. In someembodiments, the information received from the second vehicle system13504 can include physiological information, behavioral information,and/or vehicle information. As discussed above, determining a driverstate can include determining a driver state index.

At step 13516, the method includes modifying control of the firstvehicle system 13502 based on the driver state. Further, at step 13520,the first vehicle system 13502 submits information to the second vehiclesystem 13504. The information can include information about thepotential hazard, the risk level, the driver state, and the control ofthe first vehicle system. The second vehicle system 13504 receives theinformation from the first vehicle system 13502 at step 13524. Further,at step 13526, the method includes modifying control of the secondvehicle system 13504 based on the information from the first vehiclesystem 13502 and the information from the second vehicle system 13504.

Although FIG. 135A illustrates two vehicle systems in communication witheach other, more than two vehicle systems can be implemented. Forexample, FIG. 135B illustrates three vehicle systems for controlling oneor more vehicle systems in a motor vehicle. For simplicity, likenumerals in FIGS. 135A and 135B represent like elements. In FIG. 135Binformation from all three vehicle systems can be used to modify controlof the one or more vehicle systems. For example, at step 13514, thedriver state can be based on information from the first vehicle system13502, the second vehicle system 13504 and/or the third vehicle system13506. Further, at step 13520, in addition to submitting information tothe second vehicle system 13504, the method can include submittinginformation to the third vehicle system 13506.

At step 13528, the method can also include submitting information fromthe second vehicle system 13504 to the third vehicle system 13506. Atstep 13530, the method includes receiving information from the thirdvehicle system 13506. At step 13532, the method includes receivinginformation from the first vehicle system 13502 and/or the secondvehicle system 13504. At step 13534, the method includes modifyingcontrol of the third vehicle system 13506 based on information from thefirst vehicle system 13502 and/or the second vehicle system 13504. It isappreciated that the communication processes discussed in FIGS. 135A and135B can be used for any of the methods and systems discussed herein formodifying control of vehicle systems.

FIGS. 136A, 136B, 137A, and 137B are illustrative examples ofcontrolling one or more vehicle systems in response to potential hazardsin situations where the vehicle systems can be in direct communicationwith one another (e.g., intra-vehicle communication). More specifically,FIGS. 136A, 136B, 137A, and 137B illustrate exemplary embodiments ofvarious operating modes of the blind spot indicator system 224 (FIG. 2)and the electronic power steering system 132 (FIG. 2). Referring now toFIG. 136A, in this embodiment, the motor vehicle 100 is traveling on aroadway 13602. The blind spot indicator system 224 can be used tomonitor any objects traveling within a blind spot monitoring zone 13604.For example, in the current embodiment, the blind spot indicator system224 can determine that no object is inside of the blind spot monitoringzone 13604. In particular, a target vehicle 13606 is just outside of theblind spot monitoring zone 13604. In this case, no alert is sent to thedriver 102.

In FIG. 136B, to change lanes, a driver 102 can turn the wheel 134(e.g., a touch steering wheel 134). In this situation, with a driver 102fully alert, the blind spot monitoring zone 13604 has a default sizeappropriate to the amount of awareness of an alert driver. Since thetarget vehicle 13606 is not inside the blind spot monitoring zone 13604in FIG. 136B, no warnings are generated and the driver 102 has completefreedom to steer the motor vehicle 100 into the adjacent lane.

Referring now to FIGS. 137A and 137B, the motor vehicle 100 is showndriving on a roadway 13702. As the driver 102 becomes drowsy, as shownschematically in FIGS. 137A and 137B, the size of a blind spotmonitoring zone 13704 (e.g., the blind spot monitoring zone 13604) isincreased. At this point, a target vehicle 13706 is now in the enlargedmonitoring zone 13704, which results in a warning 13708, generated bythe blind spot indicator system 224. Moreover, as seen in FIG. 137B, toprevent the user from turning into the adjacent lane and potentiallycolliding with the target vehicle 13706, the electronic power steeringsystem 132 can generate a counter torque 13710 to prevent the driver 102from turning the wheel 134. This counter torque 13710 can be provided ata level to match the torque applied by the driver 102, in an opposingdirection, so that the net torque on the wheel 134 is approximatelyzero. This helps keep the motor vehicle 100 from entering the adjacentlane when a target vehicle is traveling in the blind spot of the driver102. In some cases, the warning indicator 13712 can also be activated toinform a driver that vehicle control has been modified by one or morevehicle systems. Using this arrangement, the blind spot indicator system224 and the electronic power steering system 132 can operate in acoordinated manner to warn a driver of a hazard and further control thevehicle to help avoid a potential collision.

FIG. 138 illustrates an embodiment of a process of operating a blindspot indicator system and an electronic power steering system inresponse to driver state. In some embodiments, some of the followingsteps could be accomplished by a response system 12900 of a motorvehicle. In some cases, some of the following steps can be accomplishedby an ECU 12902 of a motor vehicle. In other embodiments, some of thefollowing steps could be accomplished by other components of a motorvehicle, such as vehicle systems 126. In still other embodiments, someof the following steps could be accomplished by any combination ofsystems or components of the vehicle. It will be understood that in someembodiments one or more of the following steps can be optional. Forpurposes of reference, the following method discusses components shownin FIG. 129.

In step 13802, the ECU 12902 can receive object information. The objectcould be a vehicle or any other object that can be tracked. In somecases, for example, the object could be pedestrian or biker. In step13804, ECU 12902 can detect a potential hazard. Next, in step 13806, theECU 12902 can determine if the object poses a hazard. A method ofdetermining if an object poses a hazard for a vehicle has been discussedabove and shown in FIGS. 106 and 107. In particular, step 10604, step10606, step 10608, and step 10610 of FIG. 106 as well as each of thesteps shown in FIG. 107 provide an exemplary method to determine if theobject poses a hazard. In some cases, the step of determining if theobject poses a hazard includes checking the driver state index of adriver as discussed and shown in FIGS. 106 and 107.

In step 13808, the ECU 12902 can determine the warning type, frequency,and intensity of an alert to warn the driver. In some cases, determiningthe warning type, frequency and intensity can proceed in a similarmanner to step 10612 and step 10614 of FIG. 106. Next, the ECU 12902 canactivate a blind spot warning indicator in step 13810, to alert a driverof a potential hazard.

In step 13812, the ECU 12902 determines if the object is still insidethe blind spot monitoring zone. This step allows for the possibilitythat a driver has observed the blind spot warning indicator and adjustedthe vehicle so that there is no longer an object in the blind spot.

If there is no longer an object in the blind spot monitoring zone, ECU12902 can return to step 13802. Otherwise, the ECU 12902 can proceed tostep 13814. In step 13814, the ECU 12902 determines the trajectory ofthe tracked object. The trajectory of the object can be determined usingany methods including remote sensing as well as GPS based methods.

In step 13816, the ECU 12902 determines the relative distance betweenthe motor vehicle and the tracked object. In step 13818, the ECU 12902determines if a crash is likely between the vehicle and the trackedobject. If not, the ECU 12902 returns to step 13812 to continuemonitoring the tracked object. Otherwise, the ECU 12902 proceeds to step13820 to determine the type of power steering control to be used to helpprevent the driver from changing lanes.

In parallel with step 13820, the ECU 12902 can determine driver stateindex 13822 and use look-up table 13824 to select the appropriate typeof control. For example, if the driver state index is 1 or 2, meaningthe driver is relatively alert, no control is performed since it isassumed a driver will be aware of the potential threat posed by theobject. If the driver state index has a value of 3, meaning the driveris somewhat drowsy, some partial steering feedback is provided to helpresist any attempt by the user to turn the vehicle into the adjacentlane with the tracked object. If the driver state index has a value of4, meaning the driver is very drowsy, full steering feedback is providedto substantially prevent the driver from moving into the adjacent lane.

After the power steering control type has been selected, the ECU 12902can control the power steering system accordingly in step 13826. In somecases, at step 13828, the ECU 12902 can also activate a control warningto alert the driver that one or more vehicle systems are assisting withvehicle control.

FIG. 139 illustrates a schematic view of a further operating mode of theblind spot indicator system 224 and a brake control system. It should beunderstood that the brake control system could be any vehicle systemwith braking functions controlled by the ECU 12902. For example, thebrake control system can include, but is not limited to, an electronicstability control system 202, an antilock brake system 204, a brakeassist system 206, an automatic brake prefill system 208, a low speedfollow system 212, an automatic cruise control system 216, a collisionwarning system 218, or a collision mitigation braking system 220.

In the illustrated embodiment, the blind spot indicator system 224includes provisions for cross-traffic alert, as is known in the art,that detects objects in the blind spot during normal driving and objectsapproaching from the sides of the vehicle (i.e., cross-traffic) when thevehicle is moving forward or reverse direction. For exemplary purposes,FIGS. 138 and 139 will be described with reference to cross-traffic whenthe vehicle is in a reverse gear (i.e., when reversing out of a parkingspot). However, it is appreciated that the systems and methods describedherein can also be applicable to cross-traffic in front of the vehiclewhen the vehicle is moving in a forward direction.

Referring now to FIG. 139, the motor vehicle 100 is illustrated in aparking situation 13902 where the blind spot indicator system 224 andthe brake control system, alone or in combination, can be used toimprove a cross-traffic alert process. The blind spot indicator system224 is used to monitor any objects, for example, a first target vehicle13904 and/or a second target vehicle 13906, traveling (i.e., approachingfrom the sides of the motor vehicle 100) within a blind spot monitoringzone 13908. As discussed above, it is understood that the blind spotmonitoring zone 13908 can also be located in front of the motor vehicle100 for monitoring objects approaching from the sides of the motorvehicle 100 when the motor vehicle 100 in a forward direction. It isappreciated that the blind spot indicator system 224 can also includethe functions described above with respect to FIGS. 135-138. Forexample, the blind spot monitoring zone 13908 can increase or decreasein size based on the amount of awareness of a driver of the motorvehicle 100. Moreover, it is appreciated that the motor vehicle 100 canbe traveling in reverse or forward at an angle (e.g., a parking angle)rather than a 90 degree angle as shown in FIG. 139.

FIG. 140 illustrates an embodiment of a process of operating a blindspot indicator system including cross-traffic alert with a brake controlsystem. In some embodiments, some of the following steps could beaccomplished by a response system 12900 of a motor vehicle. In somecases, some of the following steps can be accomplished by an ECU 12902of a motor vehicle. In other embodiments, some of the following stepscould be accomplished by other components of a motor vehicle, such asvehicle systems 126. In still other embodiments, some of the followingsteps could be accomplished by any combination of systems or componentsof the vehicle. It will be understood that in some embodiments one ormore of the following steps can be optional. For purposes of reference,the following method discusses components shown in FIG. 78.

In step 14002, the ECU 12902 can receive object information. The objectcould be a vehicle or any other object that can be tracked. In somecases, for example, the object could also be pedestrian or biker. Withregards to a cross-traffic alert system, the object can be a vehicle(i.e., a first and second target vehicle 13904, 13906) in the potentialpath of a vehicle put in reverse gear. In step 14004, ECU 12902 candetect a potential hazard. Next, in step 14006, the ECU 12902 candetermine if the object poses a hazard. A method of determining if anobject poses a hazard for a vehicle has been discussed above and shownin FIGS. 106 and 107. In particular, step 10604, step 10606, step 10608,and step 10610 of FIG. 106 as well as each of the steps shown in FIG.107 provide an exemplary method to determine if the object poses ahazard. In some cases, the step of determining if the object poses ahazard includes checking the driver state index of a driver as discussedand shown in FIGS. 106 and 107.

In step 14008, the ECU 12902 can determine the warning type, frequency,and intensity of an alert to warn the driver. In some cases, determiningthe warning type, frequency and intensity can proceed in a similarmanner to step 10612 and step 10614 of FIG. 106. Next, the ECU 12902 canactivate a blind spot warning indicator in step 14010, to alert a driverof a potential hazard.

In step 14012, the ECU 12902 determines if the object is still insidethe blind spot monitoring zone. This step allows for the possibilitythat a driver has observed the blind spot warning indicator and adjustedthe vehicle so that there is no longer an object in the blind spot.

If there is no longer an object in the blind spot monitoring zone, ECU12902 can return to step 14002. Otherwise, the ECU 12902 can proceed tostep 14014. In step 14014, the ECU 12902 determines the trajectory ofthe tracked object. The trajectory of the object can be determined usingany methods including remote sensing as well as GPS based methods. Thetrajectory can also be based on a parking angle relative to the vehicleand the object, when the vehicle is put in a reverse gear and is nottravelling at a 90 degree angle.

In step 14016, the ECU 12902 determines the relative distance betweenthe motor vehicle and the tracked object. In step 14018, the ECU 12902determines if a crash is likely between the vehicle and the trackedobject. If not, the ECU 12902 returns to step 14012 to continuemonitoring the tracked object. Otherwise, the ECU 12902 proceeds to step14020 to determine the type of brake control to be used to help preventthe driver from collision with the tracked object.

In parallel with step 14020, the ECU 12902 can determine driver stateindex 14022 and use look-up table 14024 to select the appropriate typeof brake control. For example, if the driver state index is 1 or 2,meaning the driver is relatively alert, no control is performed since itis assumed a driver will be aware of the potential threat posed by theobject. If the driver state index has a value of 3, meaning the driveris somewhat drowsy, some partial brake control is provided to assist thedriver. If the driver state index has a value of 4, meaning the driveris very drowsy, full brake control provided to substantially prevent thedriver from moving into the cross-traffic. Brake control can include,but is not limited to, increasing or decreasing breaking pressure, orpre-charging or prefilling the brakes.

After the brake control type has been selected, the ECU 12902 cancontrol the brake control system accordingly in step 14026. In somecases, at step 14028, the ECU 12902 can also activate a control warningto alert the driver that one or more vehicle systems are assisting withvehicle control.

It will be appreciated that the exemplary operational response andintra-vehicle communication of one more vehicle systems can also applyto methods and systems utilizing a plurality of driver states and acombined driver state. Thus, the driver state index discussed in theexemplary operational response and intra-vehicle communication can besubstituted with more than one driver state and/or a combined driverstate index as determined by the methods and systems discussed inSection III.

FIGS. 131-135A, 135B discussed above, generally illustrate provisionsfor intra-vehicle communication and control and modifying variousdifferent vehicle systems in response to driver state based on one ormore of a hazard, a risk level, a driver state and information fromdifferent vehicle systems. These embodiments provide for varied controlof the vehicle and vehicle systems. As mentioned above, in someembodiments, the processes described above for controlling one or morevehicle systems can be used to provide semi-autonomous or fullyautonomous control to the motor vehicle. In some embodiments, thesemi-autonomous or fully autonomous controls provide intuitiveconvenience controls to the driver. In other embodiments, thesemi-autonomous or fully autonomous controls provide safety controls(e.g., to avoid potential collisions and/or hazards) to the driver. Itis understood that any of the systems and methods described above fordetermining driver states and modifying control of vehicle systems canbe implemented in whole or in part with the systems and methodsdescribed herein.

The exemplary systems and methods discussed herein related to automaticcontrol of vehicle systems, could in some embodiments, include adetermination and/or check for an auto control mode status. As discussedabove, the motor vehicle 100 can include a vehicle mode selector system238 that modifies driving performance according to preset parametersrelated to the mode selected. In one embodiment, the modes provided bythe vehicle mode selector system 238 include an auto control modestatus. The auto control mode status can be managed, activated and/ordeactivated via the vehicle mode selector system 238 and provides for asemi and/or fully automatic (e.g., autonomous) control of vehiclesystems. In some embodiments, the auto control mode can be activatedand/or deactivated by the driver. Accordingly, the driver has control asto whether automatic control of the vehicle systems can occur. In otherembodiments, the auto control mode can be automatically activated by oneor more vehicle systems, for example, based on driver state. Althoughnot every method and system discussed herein provides for adetermination and/or check for an auto control mode status, it isappreciated that the methods and systems discussed herein can allow forsuch determination and/or check.

Referring now to FIG. 141, an embodiment of a process for controllingone or more vehicle systems including auto control is illustrated. Insome embodiments, some of the following steps could be accomplished by aresponse system 12900 of the motor vehicle 100. In some cases, some ofthe following steps can be accomplished by an ECU 12902 of the motorvehicle 100. In other embodiments, some of the following steps could beaccomplished by other components of a motor vehicle, such as vehiclesystems 126. In still other embodiments, some of the following stepscould be accomplished by any combination of systems or components of thevehicle. It will be understood that in some embodiments one or more ofthe following steps can be optional.

At step 14102, the method includes receiving monitoring information. Forexample, the ECU 12902 can receive monitoring information from one ormore vehicle systems 126 and/or monitoring systems 300. As discussedabove, monitoring information can include physiological information,behavioral information and vehicular-sensed information, from variousvehicle systems 126 and/or monitoring systems 300. At step 14104, themethod includes detecting a potential hazard based on the monitoringinformation. In some embodiments, more than one potential hazard can bedetected at step 14104. In some embodiments, the ECU 12902 can detectthe hazard based on information provided by one or more vehicle systems126 and/or monitoring systems 300 (e.g., based on monitoring informationfrom step 14102). In some embodiments, the hazard is referred to as avehicular state. As an illustrative example, the ECU 12902 can receiveinformation from the blind spot indicator system 224 indicating that atarget vehicle is traveling in the blind spot monitoring zone of themotor vehicle 100. In this situation, the ECU 12902 identifies thetarget vehicle as a potential hazard. It is understood that any of thesystems and methods for detecting a potential hazard discussed abovewith FIGS. 131-138 can be implemented. In some embodiments discussedherein, step 14104 is optional. Further, in other embodiments, step14104 could be performed after step 14110.

At step 14106, the method includes determining a risk level associatedwith the potential hazard. In other words, in step 14106, the ECU 12902determines how much of a risk a potential hazard poses. It is understoodthat any of the systems and methods for determining a risk leveldiscussed above with FIGS. 131-138 can be implemented. Further, in someembodiments, step 14106 is optional. In other embodiments, step 14106could be performed after step 14110.

At step 14108, the method includes determining an auto control status.For example, the ECU 12902 can receive information from the vehicle modeselector system 238 (e.g., at step 14102) to determine if an autocontrol status is set to ON. If the auto control status is determined tobe ON, semi and/or full autonomous control of the motor vehicle 100and/or one or more vehicle systems 126 is enabled. In some embodiments,step 14108 is optional. Further, in other embodiments, step 14108 couldbe performed after step 14110.

At step 14110, the ECU 12902 can determine a driver state and/or driverstate index based on the monitoring information. The driver state can bedetermined in various ways as discussed in Section IV. In someembodiments, the driver state is based on monitoring information fromone or more vehicle systems 126 and/or monitoring systems 300. Thedriver state, in some embodiments, characterizes the attentiveness(e.g., alertness) of the driver in relation to the potential hazarddetected. In some embodiments, determining a driver state can alsoinclude determining if the driver is distracted and/or drowsy.

In one embodiment, at step 14110, the ECU 12902 can use the driver stateand/or driver state index to determine a control type (e.g., a systemstatus) as discussed in FIG. 132 at step 13214 using the look-up table13216. Other exemplary control types will be discussed herein withreference to FIGS. 143A, 143B, 143C, and 143D. At step 14112, the ECU12902 modifies control of one or more vehicle systems based at least inpart on the driver state and/or the driver state index. In someembodiments, one or more vehicle systems are modified based at least inpart on the driver state and/or the driver state index, the potentialhazard, the risk level, and/or the auto control status. Further, thevehicle systems can be modified at step 14112 based on a control typeand/or system status selected according to the driver state.

Another embodiment of a process for controlling one or more vehiclesystems in a motor vehicle including auto control is shown in FIG. 142.At step 14202, the method includes receiving monitoring information. Forexample, the ECU 12902 can receive monitoring information from one ormore vehicle systems 126 and/or monitoring systems 300 as discussedabove with FIG. 141 at step 14102. At step 14204, the method includesdetermining if an auto control status is set to ON. For example, the ECU12902 can receive information from the vehicle mode selector system 238(e.g., at step 14202) to determine if an auto control status is set toON. If the auto control status is set to ON, semi and/or full autonomouscontrol of the motor vehicle 100 and/or one or more vehicle systems 126is enabled. It is understood that in some embodiments, step 14204 isoptional. If it is determined that the auto control status is set toOFF, the method can return to step 14202. If it is determined that theauto control status is set to ON, the method proceeds to step 14206.

At step 14206, the ECU 12902 determines a driver state and/or a driverstate index. The driver state can be determined in any of the variousways discussed in Section IV. In some embodiments that will be discussedin further detail herein, the driver state is based on monitoringinformation from one or more vehicle systems 126 and/or monitoringsystems 300. In some embodiments, determining a driver state can alsoinclude determining if the driver is distracted and/or drowsy.

At step 14208, the method includes modifying control of one or morevehicle systems. For example, the ECU 12902 can modify one or morevehicle systems based on the driver state and/or driver state index. Insome embodiments, control can be based on a look-up table, for examplelook-up table 14210. More specifically, the system status and/or controlparameters of the one or more vehicle systems are modified based on thedriver state. For example, if the driver state index is 1 or 2, avehicle system can be set to a system status of no change or standardcontrol. In some embodiments, where the driver state index is 1 or 2, avehicle system can be set to a system status of auto control. If thedriver state index is 3, the vehicle system can be set to a systemstatus of some change, partial control or semi-auto control. If thedriver state index is 4, the vehicle system can be set to a systemstatus of more change, full control, or auto control. It is understoodthat the method shown in FIG. 142 can include other steps, for example,those shown in FIG. 141 (e.g., detecting a potential hazard, determininga risk level).

FIGS. 143A, 143B, 143C, and 143D illustrate exemplary look-up tables forstatus control based on a driver state index for various vehiclesystems. It is appreciated that these look-up tables are exemplary innature and other look-up tables discussed herein as well as other typesof vehicle systems and status controls can be implemented. As indicatedin FIG. 143A by look-up table 14302, a control status for a low speedfollow system can be selected according to driver state. If the driverstate index is 1 or 2, the low speed follow system 212 status is set tostandard. If the driver state index is 3 or 4, the low speed followsystem 212 status is set to auto. It is appreciated that in someembodiments, which are described herein, when the auto control status isset to ON, and the driver state index is 1 or 2 (e.g., the driver isattentive), the low speed follow system 212 status can be set to auto toallow for autonomous control of the low speed follow system 212 when thedriver is attentive. Further, in other embodiments, the low speed followsystem 212 status can be set to ON or OFF based on the driver state(e.g., FIG. 96, look-up table 9610).

As indicated in FIG. 143B by look-up table 14304, a control status for alane keep assist system based can be selected according to a driverstate. If the driver state index is 1 or 2, the lane keep assist system226 status is set to standard. If the driver state index is 3 or 4, thelane keep assist system 226 status is set to auto. It is appreciatedthat in some embodiments, which are described herein, when the autocontrol status is set to ON, and the driver state index is 1 or 2 (e.g.,the driver is attentive), the lane keep assist system 226 status can beset to auto to allow for autonomous control of the lane keep assistsystem 226 when the driver is attentive. In some embodiments, the lanekeep assist system 226 can vary from standard to low control based onthe driver state (e.g., FIG. 102, look-up table 10218).

As indicated in FIG. 143C by look-up table 14306, a control status foran automatic cruise control system can be selected according to a driverstate. If the driver state index is 1, the automatic cruise controlsystem 216 status can be set to manual or OFF thereby requiring a manualswitch/button input to modify a headway distance. If the driver stateindex is 2, a headway distance (e.g., control parameter) of theautomatic cruise control system 216 can set to a minimum gap. If thedriver state index is 3 or 4, a headway distance (e.g., controlparameter) of the automatic cruise control system 216 can set to amaximum gap. It is appreciated that in some embodiments, which aredescribed herein when the auto control status is set to ON, and thedriver state index is 1 or 2 (e.g., the driver is attentive), theautomatic cruise control system 216 can be set to auto to allow forautonomous control of the automatic cruise control system 216 while thedriver is attentive. In other embodiments, the automatic cruise controlsystem 216 can be set to ON or OFF and/or a distance setting can be setin accordance with the driver state (e.g., FIG. 94, look-up tables 9408,9420).

In another embodiment, modifying control of the one or more vehiclesystems can include activating a visual indicator (e.g., visual devices140) based on the driver state and the control type of the vehicleand/or vehicle systems. As an illustrative example, if the motor vehicle100 and/or one or more vehicle systems 126 and the driver 102 is notdistracted and/or drowsy, the light bar 1808 of the touch steering wheel1802 (see FIG. 18) can be activated to emit a green colored lightthereby indicating the auto control status and driver state to thedriver 102. As another illustrative example, if the motor vehicle 100and/or one or more vehicle systems 126 is in an auto control mode andthe driver 102 is distracted and/or drowsy, the light bar 1808 of thesteering wheel 1802 (see FIG. 18) can be activated to emit a red coloredlight thereby indicating the auto control status and driver state to thedriver 102. In further example, if the motor vehicle 100 and/or one ormore vehicle systems 126 is in an auto control mode with partial control(e.g., semi-autonomous control) and the driver 102 is not distractedand/or drowsy, the light bar 1808 of the steering wheel 1802 (See FIG.18) can be activated to emit a partially green colored light therebyindicating the auto control mode and driver state to the driver.

As indicated in FIG. 143D by look-up table 14308, a control status forvisual devices can be selected according to a driver state. In any ofthe above examples, when the light bar 1808 of the steering wheel 1802is activated to emit a color, the response system 12900 can flash thelight. For example, flash the red light to get the driver's attention.In addition, in any of the above examples, when the light bar 1808 ofthe steering wheel 1802 is activated to emit a color, the responsesystem 12900 can control audio devices 144 to provide an audible sound.For example, when the light bar 1808 of the steering wheel 1802 isactivated to emit a red colored light, the audio devices 144 can beactivated to provide an audible sound indicating the auto control statusand driver state to the driver 102. Any color or sound combinations canbe used.

In some embodiments, control of vehicle systems 126, including vehiclesystem warnings, can be activated and/or deactivated based the driverstate. For example, if the driver 102 is attentive (e.g., alert, aware)of potential hazards surrounding the motor vehicle 100, some vehiclesystems 126 and warnings can be deactivated (e.g., turned OFF).Accordingly, the driver 102 is given full control of the motor vehicle100 and unnecessary warnings are suppressed since the driver 102 isattentive of any potential hazards. Referring now to FIG. 144 a flowchart is shown of an embodiment for controlling one or more vehiclesystems including suppressing and/or restricting vehicle systems andwarnings. At step 14402, the method includes the ECU 12902 receivingmonitoring information from one or more vehicle systems 126 and/or oneor more monitoring systems 300. At step 14404, the method includesdetermining if a potential hazard exists based on the monitoringinformation. If a potential hazard does not exist, the method can returnto step 14402.

If a potential hazard does exist, the method proceeds to step 14406. Atstep 14406, the ECU 12902 can determine a driver state and/or driverstate index. The driver state index is based on the monitoringinformation received at step 14402. The driver state index can be basedon information from one or more vehicle systems 126 and/or one or moremonitoring systems 300. At step 14408, the method includes determiningif the driver is distracted based on the driver state index. If thedriver not distracted (e.g., aware of the potential hazard, alert,attentive), at step 14410, the method includes modifying control of oneor more vehicle systems. More specifically at step 14410, the systemstatus of one or more vehicle systems 126 can be set to no control orturned OFF (e.g., disabled). In another embodiment, at step 14410, thesystem status of one or more vehicle systems 126 can be set to autocontrol. Accordingly, modifying control of one or more vehicle systemsat step 11410 can include suppressing one or more vehicle systems 126and/or vehicle system warnings that would normally be triggered by thevehicle systems 126 based on the potential hazard. Further, modifyingcontrol of one or more vehicle systems 126 can include deactivatingvehicle systems 126 and/or functions that would normally be triggered bythe vehicle systems 126 based on the potential hazard. For example, thelane keep assist system 226 can be disabled (e.g., turned OFF) at step14410 so that steering assistance is not provided, thereby allowing thedriver 102 to have full control of steering.

If the driver is distracted, at step 14412, the method includesmodifying control of one or more vehicle systems 126. More specifically,at step 14412, modifying control of one or more vehicle systems 126 caninclude activating warnings of certain systems based on the potentialhazard. Additionally, modifying control of one or more vehicle systems126 at step 14412 can include setting a control parameter and/or asystem status of one or more vehicle systems 126. For example, thesystem status of the lane keep assist system 226 can be set to standardat step 14412. In another embodiment, the system status of the lane keepassist system 226 can be set to auto.

Another embodiment of a process for controlling one or more vehiclesystems including confirming a risk and/or hazard is shown in FIG. 145.It is appreciated that the method shown in FIG. 145 could be implementedwith any of the illustrative examples described above and with anyvehicle systems 126 or monitoring systems 300 discussed previously. Asdiscussed above, in some embodiments, although a hazard or risk ispresent, the driver may be aware of the hazard and risk. In thesesituations, the one or more vehicle systems 126 can be modified toaccount for confirmation of the potential hazard and/or risk by thedriver 102. FIG. 145 illustrates a general method of confirming apotential hazard and modifying one or more vehicles based on theconfirmation.

At step 14502, the ECU 12902 can receive monitoring information from oneor more vehicle systems 126 and/or one or more monitoring systems 300 asdescribed in detail above. For example, the ECU 12902 can receivephysiological information, behavioral information and vehicleinformation. At step 14504, the ECU 12902 can detect a potential hazardas described in detail above based on the monitoring informationreceived at step 14502. At step 14506, the ECU 12902 can determine arisk level, for example, based on the probability that the vehicle willencounter the hazard. It is understood that in some embodiments, step14506 is optional and/or can be determined after step 14508.

At step 14508, the ECU 12902 determines if the potential hazard has beenconfirmed by the driver. Said differently, it is determined if thedriver 102 is aware (e.g., attentive, alert) of the potential hazard. Inanother embodiment, a risk level may be determined, and at step 14508,the method determines if the risk presented by the potential hazard hasbeen confirmed by the driver 102. To determine if the potential hazardhas been confirmed, at step 14510 the method can include determining adriver state and/or driver state index based on the monitoringinformation. The driver state can be based on monitoring informationfrom vehicle systems and/or monitoring systems, for example, themonitoring information received at step 14502. Further, the driver statecan be based on a plurality of driver states. In some embodiments, thedriver state determined at step 14510 is based on an analysis ofmonitoring information relative to the potential hazard.

As discussed above, in some embodiments, step 14506 can includedetermining if a risk level is high. Thus, in one embodiment,determining if the potential hazard is confirmed at step 14508 can alsobe based on the risk level. Accordingly, if the risk level is high, evenif it is determined that the driver state is attentive at step 14510,the ECU 12902 can determine the potential hazard is not confirmed basedon a high risk level. Thus, even if the driver 102 is aware of thepotential hazard, if the risk level of the potential hazard is high, itis determined at step 14508 that the potential hazard is not confirmed.

If the potential hazard is not confirmed, at step 14512, the ECU 12902modifies the control of one or more vehicle systems. More specifically,at step 14512, modifying control of one or more vehicle systems 126 caninclude activating warnings of certain vehicle systems 126 based on thepotential hazard. Thus, modifying control of one or more vehicle systems126 can include setting a control status of the one or more vehiclesystems 126 to standard control or auto control.

If the potential hazard has been confirmed indicating that the driver102 is aware of the potential hazard, at step 14514, the method includesmodifying one or more vehicle systems. More specifically, at step 14514,if the potential hazard has been confirmed, vehicle systems and/orvehicle system warnings can be deactivated and/or overridden. Saiddifferently, modifying control of one or more vehicle systems at step14514 can include suppressing vehicle system warnings and/or functionsthat would normally be triggered by the vehicle systems based on thepotential hazard. Thus, modifying control of one or more vehicle systems126 can include setting a control status to no control or disabled(e.g., OFF).

A specific example will now be described with reference to FIG. 145. Atstep 14502, monitoring information is received from one or more vehiclesystems 126 and/or one or more monitoring systems 300. For example,monitoring information can be received from a blind spot indicatorsystem 224. At step 14504, the blind spot indicator system 224 candetect a hazard as an object in a blind spot monitoring zone of themotor vehicle 100. The blind spot indicator system 224 can determine ifthe hazard poses a risk based on the methods described above at step14506. In some embodiments, at step 14506, determining if the hazardposes a risk also includes determining if the risk level is high.

At step 14508, the ECU 12902 can determine if the potential hazard isconfirmed. In some embodiments, the determination at step 14508 is basedon the monitoring information and a driver state and/or a driver stateindex determined at step 14510. For example, the ECU 12902 can receivehead movement information (e.g., a head look) at step 14502 from a headmovement monitoring system 334 and/or eye gaze information from aneye/facial movement monitoring system 332. Further, the ECU 12902 canreceive information about a potential lane departure from a lanedeparture warning system 222 at step 14502.

Based on this information, the ECU 12902 determines a driver stateand/or driver state index at step 14510. The driver state can be basedon an analysis of the monitoring information (e.g., head movement, eyegaze, potential lane departure direction) relative to the potentialhazard. In this example, the ECU 12902 may determine that a potentiallane departure is in the same direction as the object (e.g., targetvehicle) and blind spot monitoring zone, but the head look and/or eyegaze of the driver indicates the driver 102 is looking at the object(e.g., target vehicle) and the blind spot monitoring zone. Thus, thedriver 102 is aware (e.g., alert, attentive) of the potential hazard.Accordingly, the ECU 12902 can determine driver state as attentive atstep 14510, and at step 14508, the ECU 12902 determines the potentialhazard is confirmed and the method can proceed to step 14514.

In this example, at step 14514, the ECU 12902 can disable (e.g., turnOFF) the lane departure warning system 222 and/or the blind spotindicator system 224. Accordingly, warnings typically emitted by thesesystems will be suppressed. In another example, the ECU 12902 can set acontrol type of the lane keep assist system 226 to no control (e.g.,disabled, turn OFF) so that no power steering assistance is provided.

In another illustrative example, the ECU 12902 can detect turn signalinformation from a turn signal control system 240 at step 14502. Basedon this information and other monitoring information, at step 14510, theECU 12902 determines a driver state and/or driver state index. In thisexample, the ECU 12902 may determine that a potential lane departure isin the same direction as the object and blind spot monitoring zone, butthe turn signal information indicates a turn signal has been activatedtoward the object and the blind spot monitoring zone. In addition, thehead look and/or eye gaze of the driver 102 indicates the driver hasconfirmed the potential hazard. Accordingly, the driver state isdetermined to be attentive at step 14510 and at step 14508 it isdetermined that the driver confirmed the potential hazard. However, inanother embodiment, even if the driver state is determined to beattentive at step 14510, if a risk level is determined and the risklevel is determined to be high, the ECU 12902 may determine thepotential hazard is not confirmed at step 14508.

If the potential hazard is confirmed at step 14508, at step 14514, oneor more vehicle systems are modified based on the driver state. Forexample, the ECU 12902 may turn off warnings from the lane departurewarning system 222 and may turn off the lane keep assist system 226.Accordingly, in this example, since the potential hazard has beenconfirmed, the ECU 12902 will modify the vehicle systems to allow thedriver to continue with a potential lane departure and possibly changelanes (squeeze in front of the vehicle in the blind spot monitoringzone). If it is determined that the potential hazard is not confirmed atstep 14508, the driver is determined to be distracted and the lanedeparture warning system 222 and lane keep assist system 226 willoperate to prevent the vehicle 100 from completing the lane change orwarn the driver of the vehicle in the blind spot monitoring zone.

It is understood that in some embodiments, the modification oradjustment of one or more vehicle systems can be modified and/oradjusted again (e.g., back to an original state) based on a change indriver state. For example, in some embodiments where the ECU 12902deactivates and/or turns OFF any vehicle systems 126, the ECU 12902 canautomatically reactivate and/or turn ON these vehicle systems upondetecting a change in driver state, for example a driver state that isdistracted and/or drowsy. In other embodiments, the ECU 12902 canautomatically check and/or determine a driver state at a predeterminedtime interval to determine if there is a change in driver state and thevehicle systems 126 should be modified again (e.g., reverted to anoriginal status/state). Thus, in some examples, vehicle systems can beenabled and disabled within seconds based on the driver state. As anillustrative example, if the ECU 12902 determines the driver state asattentive and disables (e.g., turn OFF) the lane departure warningsystem 222 (e.g., suppressing warnings), the ECU 12902 can subsequentlyreactivate (e.g., enable, turn ON) the lane departure warning system 222when the ECU 12902 determines the driver state is distracted.

The exemplary operational responses of the one or more vehicle systemsdescribed above can be implemented with the methods and systems fordetermining one or more driver states, determining a combined driverstate, confirming one or more driver states, determining a vehicularstate, as discussed above. Specific examples of controlling vehiclesystems according to the methods of FIGS. 141, 142, 144 and 145 will nowbe described. These examples are exemplary in nature and it isunderstood that other vehicle systems and combinations of vehiclesystems can be implemented. Further, it is understood that somecomponents of FIGS. 141, 142, 144 and 145 can be omitted and/orrearranged into other configurations. In some embodiments, vehiclesystems are modified for semi and/or automatic control based on a driverstate where the driver state is determined relative to a potentialhazard. In other embodiments, vehicle systems are modified for semiand/or automatic control based on a combined driver state. The combineddriver state can be based on different types of behavioral informationand vehicular-sensed information. Some of the controls and/ormodifications of vehicle systems provide intuitive driver controlsand/or convenience features allowing control customized to the driverand the driver state.

Referring now to FIG. 146, a method for operating a lane departurewarning system in response to driver state is illustrated. In someembodiments, some of the following steps could be accomplished by aresponse system 12900 of the motor vehicle 100. In some cases, some ofthe following steps can be accomplished by an ECU 12902 of a motorvehicle. In other embodiments, some of the following steps could beaccomplished by other components of a motor vehicle, such as vehiclesystems 126. In still other embodiments, some of the following stepscould be accomplished by any combination of systems or components of thevehicle. It will be understood that in some embodiments one or more ofthe following steps can be optional.

At step 14602, the method includes the ECU 12902 receiving informationfrom the lane departure warning system 222 (e.g., monitoringinformation). At step 14604, the ECU 12902 determines if a potentiallane deviation exists with respect to the motor vehicle 100 (e.g., apotential hazard) based on the information from the lane departurewarning system 222. In some embodiments, the potential lane deviationcan be determined based on lane departure warning information asdiscussed above with FIGS. 100 and 101. If a potential lane deviationdoes exist, the method can proceed to step 14606. Otherwise, the methodcan proceed back to step 14602.

At step 14606, the ECU 12902 receives head movement information, from,for example, the head movement monitoring system 334, and/or eye gazeinformation from the eye/facial movement monitoring system 332. In someembodiments, the head movement and/or eye gaze information can bereceived at step 14602. The head movement information can includeinformation about a head pose and a head look of the driver as discussedabove in Section III (B) (2) and with FIGS. 16A, 16B and 17. Thus, atstep 14608, the ECU 12902 can analyze the head movement (e.g., headlook) and/or the eye gaze relative to the potential hazard, for example,the potential lane deviation. More specifically, the ECU 12902determines if a head look and/or eye gaze of the driver 102 is directedtoward the potential lane deviation.

Accordingly, at step 14610, the method includes determining a driverstate and/or driver state index. For example, the driver state isdetermined based on the monitoring information relative to the potentialhazard. In some embodiments, step 14610 can also include determining ifthe driver is attentive and/or distracted based on the driver stateand/or driver state index. More specifically, in FIG. 146, the ECU 12902determines the driver state based on at least the head movementinformation and/or eye gaze information received at step 14606 and theanalysis of the head movement and/or eye gaze information relative tothe lane deviation at step 14608. Said differently, the driver stateand/or the driver state index is based at least in part on the headmovement and/or eye gaze information and the potential lane deviation.

Thus, in one embodiment, if the head look is a forward-looking headlook, the driver state index is determined to be low (e.g., attentive)at step 14610. Similarly, if the head look is directed in the samedirection of the potential lane deviation, the driver state index isdetermined to be low (e.g., attentive) at step 14610. However, if thehead look is not forward-looking or is not directed to the samedirection as the possible lane deviation, the driver state index isdetermined to be high (e.g., not attentive) at step 14610.

Accordingly, at step 14612, the ECU 12902 modifies one or more vehiclesystems based on the driver state and/or driver state index determinedat step 14610. In one embodiment, the ECU 12902 modifies a control type(e.g., system status) of one or more vehicle systems 126. For example,if the driver state index indicates an attentive driver state, the ECU12902 can set the control type of the lane departure warning system 222to disabled and/or no control (e.g., OFF). Accordingly, the warningsemitted by the lane departure warning system 222 are deactivated and/orsuppressed. If the driver state index indicates a distracted driverstate, the ECU 12902 can set the control type of the lane departurewarning system 222 to enabled and/or standard control (e.g., ON). Forexample, the ECU 12902 can activate the warnings emitted by the lanedeparture warning system 222. In another embodiment, if the driver stateindex indicates a distracted driver state, the ECU 12902 activates thewarnings emitted by the lane departure warning system 222 and activatesa lane keep assist system 226 (e.g., system status to ON) to providelane keeping assistance.

Referring now to FIGS. 147A and 147B, a schematic view of controlling alane departure warning system according to the method of FIG. 146 isshown. In FIG. 147A, the motor vehicle 100 is travelling on a roadway14702 and is approaching a centerline 14704. The head look of the driver102 is forward-looking relative to the motor vehicle 100. Accordingly,based on the potential lane deviation of the motor vehicle 100 and thehead look of the driver 102, the ECU 12902 determines the driver stateto be attentive. Thus, the ECU 12902 modifies the lane departure warningsystem 222 by setting the system status of the lane departure warningsystem 222 to no control or disabled (e.g., OFF). Therefore, the lanedeparture warning 14706 is deactivated.

In FIG. 147B, the motor vehicle 100 is approaching the centerline 14704and the head look of the driver 102 is not forward-looking (i.e., headdown, head look down). Accordingly, the ECU 12902 determines the driverstate to be distracted and modifies the lane departure warning system222 by setting a system status to enabled and/or standard control (e.g.,ON). Therefore, the lane departure warning 14706 is activated.

Referring now to FIG. 148, a method for operating a blind spot indicatorsystem in response to driver state is illustrated. At step 14802, themethod includes the ECU 12902 receiving information from a blind spotindicator system 224 (e.g., monitoring information). At step 14804, theECU 12902 determines if a potential hazard exists based on theinformation from the blind spot indicator system 224. For example, theECU 12902 can detect a potential hazard as an object (e.g., a targetvehicle) inside a blind spot monitoring zone of the motor vehicle 100.If a potential hazard is not detected at step 14804, the method canreturn to step 14802. Otherwise, the method proceeds to step 14806.

At step 14806, the ECU 12902 receives head movement information and/oreye gaze information, for example from a head movement monitoring system334 and/or an eye/facial movement monitoring system 332. In someembodiments, the head and/or eye gaze movement information is receivedat step 14802. The head movement information can include informationabout a head pose and a head look of the driver as discussed above inSection III (B) (2) and with FIGS. 16A, 16B, 17. Thus, at step 14808,the ECU 12902 can analyze the head movement and/or eye gaze informationrelative to the target vehicle and/or blind spot monitoring zone (e.g.,the potential hazard). Said differently, the ECU 12902 can determine ahead movement (e.g., a head look) and/or eye gaze relative to thepotential hazard, for example, the blind spot monitoring zone and/or thetarget vehicle. More specifically, the ECU 12902 determines if the headlook and/or eye gaze is directed away from the blind spot monitoringzone and/or the target vehicle.

Accordingly, at step 14810, the method includes determining a driverstate and/or a driver state index. For example, the driver state isdetermined based on monitoring information relative to the potentialhazard. In some embodiments, step 14810 can also include determining ifthe driver is attentive and/or distracted based on the driver stateand/or driver state index. More specifically, in FIG. 148, the ECU 12902determines the driver state based on at least the head movement and/oreye gaze information received at step 14806 and the analysis of the headmovement and/or eye gaze relative to the target vehicle and/or blindspot monitoring zone at step 14808. Said differently, the driver stateand/or the driver state index is based at least in part on the headmovement and/or eye gaze information and the target vehicle and/or blindspot monitoring zone.

For example, if the head look or eye gaze is a forward-looking head lookor eye gaze, the driver state index is determined to be low (e.g.,attentive) at step 14810. If the head look or eye gaze is directed awayfrom the object, the blind spot monitoring zone, and/or a forward way ofthe vehicle, the driver state index is determined to be high (e.g., notattentive) at step 14810.

At step 14812, the ECU 12902 modifies one or more vehicle systems basedon the driver state and/or driver state index. In one embodiment, theECU 12902 modifies a control type of one or more vehicle systems. Forexample, if the driver state index indicates an attentive driver state,the ECU 12902 can set the control type (e.g., system status) of theblind spot indicator system 224 to disabled and/or no control (e.g.,OFF). Accordingly, the ECU 12902 deactivates warning signals emittedfrom the blind spot indicator system 224. If the driver state indexindicates a distracted driver state, the ECU 12902 can set the controltype (e.g., system status) of the blind spot indicator system 224 toenabled and partial and/or full control (e.g., ON). Thus, the ECU 12902activates the warning signals emitted from the blind spot indicatorsystem 224. In addition, if the driver state is distracted, the ECU12902 can modify the activation time of the warning signals. Forexample, the ECU 12902 can increase the activation time of the warningsignals, based in part, on the driver state and/or driver state index.

Referring now to FIGS. 149A and 149B, a schematic view of controlling ablind spot indicator system in accordance with the method of FIG. 148 isshown. In FIG. 149A, the blind spot indicator system 224 detects atarget vehicle 14902 is traveling on a road 14906 inside of a blind spotmonitoring zone 14904 of the motor vehicle 100. Here, the head lookand/or eye gaze of the driver 102 is forward-looking relative to themotor vehicle 100. Accordingly, based on the potential hazard with thetarget vehicle 14902 and the head look and/or eye gaze of the driver,the ECU 12902 determines the driver state to be attentive and controlsthe blind spot indicator system 224 by disabling the blind spotindicator system 224 and/or setting the control status of the blind spotindicator system 224 to no control (e.g., OFF). Accordingly, the blindspot indicator warning 14908 is deactivated (e.g., suppressed) by theECU 12902.

In FIG. 149B, the blind spot indicator system 224 detects the targetvehicle 14902 is traveling on the road 14906 inside of the blind spotmonitoring zone 14904 of the motor vehicle 100, but the head look and/oreye gaze of the driver 102 is directed away from the target vehicle14902 and the blind spot monitoring zone 14904. Accordingly, based onthe potential hazard with the target vehicle 14902 and the head lookand/or eye gaze of the driver 102, the ECU 12902 determines the driverstate to be distracted and controls the blind spot indicator system 224by enabling the blind spot indicator system 224 and/or setting thecontrol status of the blind spot indicator system 224 to partial and/orfull control (e.g., ON). Accordingly, the blind spot indicator systemwarning 14908 is activated by the ECU 12902.

Referring now to FIG. 150, a method for operating a blind spot indicatorsystem and a lane departure warning system based on driver state isillustrated. At step 15002, the method includes the ECU 12902 receivinginformation from the blind spot indicator system 224 (e.g., monitoringinformation). At step 15004, the ECU 12902 detects a potential hazardbased on the information from the blind spot indicator system 224. Forexample, the ECU 12902 can detect a potential hazard as an object (e.g.,a target vehicle) inside a blind spot monitoring zone. If a potentialhazard is not detected at step 15004, the method can return to step15002. Otherwise, the method proceeds to step 15006.

At step 15006, the ECU 12902 receives information from the lanedeparture warning system 222, head movement information from a headmovement monitoring system 334 and/or eye gaze information from aneye/facial movement monitoring system 332. The information from the lanedeparture warning system 222 can include information about a potentiallane deviation and a direction of the lane deviation. The head movementinformation can include information about a head pose and a head look ofthe driver as discussed above in Section III (B) (2) and with FIGS. 16A,16B and 17. It is understood that the information from the lanedeparture warning system 222, head movement information from a headmovement monitoring system 334 and/or eye gaze information from aneye/facial movement monitoring system 332 can be received at step 15002.

At step 15008, the ECU 12902 can analyze the lane departure warninginformation and head movement and/or eye gaze information relative tothe target vehicle and/or blind spot monitoring zone (e.g., thepotential hazard). Said differently, the ECU 12902 can determine adirection of a potential lane deviation and a direction of a headmovement (e.g., head look) and/or eye gaze relative to the potentialhazard, for example, the blind spot monitoring zone and/or the targetvehicle.

Accordingly, at step 15010, the method includes determining a driverstate and/or a driver state index. For example, the driver state isbased on monitoring information relative to the potential hazard. Insome embodiments, step 15010 can also include determining if the driveris attentive and/or distracted based on the driver state and/or thedriver state index. More specifically, in FIG. 150, the ECU 12902determines the driver state based on at least the lane departure warninginformation, head movement and/or eye gaze information received at step15006, and the analysis of the lane departure warning information andhead movement and/or eye gaze information relative to the potentialhazard at step 15008. Said differently, the driver state and/or thedriver state index is based at least in part on the lane departurewarning information, the head movement and/or eye gaze information andthe target vehicle and/or blind spot monitoring zone.

For example, if the head look and/or eye gaze is forward-looking and thelane departure warning system information indicates a possible lanedeviation towards the object and/or blind spot monitoring zone, thedriver state is determined to be distracted at step 15010. Similarly, ifthe head look and/or eye gaze is not towards the object and/or blindspot monitoring zone and the lane departure warning system 222information indicates a possible lane deviation towards the objectand/or blind spot monitoring zone, the driver state is determined to bedistracted at step 15010. However, if the head look and/or eye gaze isdirected to the object and/or blind spot monitoring zone and the lanedeparture warning system 222 information indicates a possible lanedeviation towards the object and/or blind spot monitoring zone, thedriver state is determined to be attentive at step 15010.

At step 15012, the ECU 12902 modifies one or more vehicle systems basedon the driver state and/or driver state index. In one embodiment, theECU 12902 modifies a control type (e.g., a system status) of one or morevehicle systems 126. For example, if the driver state is attentive, theECU 12902 can set the control type of the blind spot indicator system224 and/or the lane departure warning system 222 to disabled and/or nocontrol (e.g., OFF). Accordingly, the ECU 12902 deactivates warningsignals emitted from the blind spot indicator system 224 and/or the lanedeparture warning system 222. If the driver state index indicates adistracted driver state, the ECU 12902 can set the control type of theblind spot indicator system 224 and/or the lane departure warning system222 to enabled and partial and/or full control (e.g., ON). Thus, the ECU12902 activates the warning signals emitted from the blind spotindicator system 224 and/or the lane departure warning system 222. Inaddition, if the driver state is distracted, the ECU 12902 can modifythe activation time of the warning signals. For example, the ECU 12902can increase the activation time of the warning signals, based in part,on the driver state and/or driver state index.

Referring now to FIGS. 151A and 151B, a schematic view of controllingone or more vehicle systems in accordance with the method of FIG. 150 isshown. In FIG. 151A, the blind spot indicator system 224 detects atarget vehicle 15102 is traveling inside of a blind spot monitoring zone15104 of the motor vehicle 100, the motor vehicle 100 is approaching acenterline 15106 of a road 15108. Here, the head look of the driver 102is forward-looking relative to the motor vehicle 100. Accordingly, basedon the potential hazard, the potential lane deviation, and the head lookand/or eye gaze of the driver 102, the ECU 12902 determines the driverstate to be distracted. Thus, the ECU 12902 controls the blind spotindicator system 224 and the lane departure warning system 222 byenabling said systems and setting the control status of said systems topartial and/or full control (e.g., ON). Thus, the ECU 12902 activatesthe blind spot indicator system warning 15110 and lane departure warning15112 since the driver state is distracted.

In FIG. 151B, the blind spot indicator system 224 detects the targetvehicle 15102 is traveling inside of the blind spot monitoring zone15104 of the motor vehicle 100, the motor vehicle 100 is approaching thecenterline 15106 of the road 15108. Here, the head look of the driver102 is looking towards the blind spot monitoring zone 15104.Accordingly, based on the potential hazard, the potential lanedeviation, and the head look and/or eye gaze of the driver 102, the ECU12902 determines the driver state to be attentive. Thus, the ECU 12902controls the blind spot indicator system 224 and the lane departurewarning system 222 by disabling said systems and setting the controlstatus of said systems to no control (e.g., OFF). Accordingly, the ECU12902 deactivates the warnings 15110 and 15112 since the driver state isattentive.

Referring now to FIG. 152, a method of an embodiment of a process forcontrolling an idle mode of an engine based on driver state according toan exemplary embodiment is shown. As discussed above, the engine 104 ofthe motor vehicle 100 can include an idle stop function that iscontrolled by the ECU 12902 and/or the engine 104. Specifically, theidle stop function includes provisions to automatically stop and restartthe engine 104 to help maximize fuel economy depending on environmentaland vehicle conditions. In some embodiments, the idle stop function canbe activated based on a timer function. At step 15202, the methodincludes receiving braking information (e.g., monitoring information),from, for example, the antilock brake system 204. It is understood thatthe braking information can be received from any braking system and/orfrom the engine 104. More specifically, braking information can includeinformation from any sensors and/or vehicle systems. For example, theECU 12902 can receive information that a brake switch (e.g., brakepedal) has been applied to determine if the driver 102 is currentlybraking. In another example, the ECU 12902 can use other vehicleinformation to determine if the brake pedal is depressed, the brakepedal is released, braking is being applied, braking rate, brakingpressure, among others. In some embodiments described herein, brakinginformation can also include information about acceleration, received,for example, from the ECU 12902. For example, indication that anaccelerator switch (e.g., an accelerator pedal) has been applied,accelerator pedal input, accelerator pedal input pressure/rate, amongothers.

At step 15204, it is determined if the vehicle is stopped based on thebraking information (e.g., the vehicle is at a complete stop). If thevehicle is not at a complete stop, the method can return to step 15202.If the vehicle is at a complete stop, the method can proceed to step15206. At step 15206 it is determined if the idle mode function is setto ON. This determination can be based on the monitoring informationreceived at step 15202. For example, the monitoring information, todetermine if the idle mode function status (e.g., ON/OFF), can bereceived from the engine 104 and/or the ECU 12902. It is understood thatin some embodiments, step 15206 can be optional.

If the determination at step 15206 is NO (i.e., the idle mode functionis set to OFF), the method can return to step 15202. Otherwise, themethod proceeds to step 15208. At step 15208, the ECU 12902 receiveshand contact information indicating hand contact of the driver with thesteering wheel, for example, the touch steering wheel 134. In oneembodiment, the hand contact information can be received from the touchsteering wheel system 134 and/or the EPS system 132. In anotherembodiment, hand contact information can be received from opticalsensors and analyzed, for example, by the gesture recognition monitoringsystem 330. In some embodiments, the hand contact information can bereceived at step 15202. It is understood that steps 15208 and 15210 canbe part of determining a driver state based on behavioral information.

At step 15210, it is determined if there is hand contact with thesteering wheel based on the hand contact information. Said differently,it is determined if one or both hands are on the steering wheel 134. Ifthere is at least one hand on the steering wheel 134, the method returnsto step 15202. Otherwise, the method proceeds to step 15212 where theECU 12902 engages the idle mode function of the engine 104 (i.e., turnsthe engine OFF).

In order to disengage the idle mode function, at step 15214, the methodincludes receiving hand contact information, similar to step 15208. Atstep 15216, it is determined if one or both hands are in contact withthe steering wheel 134 based on the hand contact information. If thedetermination at step 15216 is NO (i.e., no hands on the steering wheel134), the process returns to step 15214. Otherwise, at step 15218, theECU 12902 disengages the idle mode function of the engine 104 (i.e.,turns the engine ON).

Referring now to FIG. 153, a method for controlling a brake hold featureof an electric parking brake system is shown. At step 15302, the ECU12902 receives braking information (e.g., monitoring information), from,for example, the antilock brake system 204. It is understood that thebraking information can come from any of the braking systems, from theelectric parking brake system 210 and/or from the engine 104. At step15304, the ECU 12902 determines if the vehicle is stopped based on thebraking information (e.g., the vehicle is at a complete stop). If thevehicle is not at a complete stop, the method can return to step 15302.If the vehicle is at a complete stop, the method can continue to step15306.

At step 15306, the ECU 12902 determines if the brake pedal of the motorvehicle 100 is released (e.g., not depressed) based on, for example, thebraking information received at step 15302. If the determination is NO,the method can return to step 15302. If determination is YES, the methodcan continue to step 15308. At step 15308, hand contact information isreceived indicating hand contact of the driver with the steering wheel,for example, the touch steering wheel 134. The hand contact informationcan be received by the ECU 12902 from the touch steering wheel system134 and/or the EPS system 132. In some embodiments, the hand contactinformation can be received at step 15302.

At step 15310, it is determined if there is hand contact with thesteering wheel based on the hand contact information. Said differently,it is determined if one or both hands are on the touch steering wheel134. If there is at least one hand on the steering wheel 134, the methodreturns to step 15302. Otherwise, the method proceeds to step 15312where the ECU 12902 engages the brake hold function of the electricparking brake system 210 (i.e. the vehicle 100 remains stopped withoutthe driver 102 needing to engage the brake pedal or shift to park).

In order to disengage (e.g., release) the brake hold function, at step15314, the method includes receiving braking information and/or handcontact information, similar to steps 15302 and 15308. At step 15316,the ECU 12902 determines if an accelerator pedal of the motor vehicle100 is engaged (e.g., depressed) or the brake pedal of the motor vehicle100 is engaged (e.g., depressed) based on the braking information. Ifthe determination at step 15316 is YES, the method proceeds to step15318 where the ECU 12902 disengages (e.g., releases) the brake holdfunction.

If the determination at step 15316 is NO, the method proceeds to step15320 where the ECU 12902 determines if there is hand contact with thesteering wheel based on the hand contact information. Said differently,it is determined if one or both hands are on the steering wheel 134. Ifthere is at least one hand on the steering wheel 134, the methodproceeds to step 15318. Otherwise, the method proceeds back to step15314.

Referring now to FIG. 154, a method for disengaging (e.g., releasing) anelectric parking brake system is shown. At step 15402, the methodincludes receiving electric parking brake information from the electricparking brake system 210. At step 15404, it is determined if theelectric parking brake status is set to ON based on the informationreceived at step 15402. If the determination at step 15404 is NO (i.e.,the electric parking brake status is set to OFF), the method returns tostep 15402. Otherwise, the method proceeds to step 15406.

At step 15406, the ECU 12902 receives hand contact information andbraking information. The hand contact information can be received fromthe touch steering wheel system 134 and/or the EPS system 132. Thebraking information can be received, for example, from the antilockbrake system 204. It is understood that in some embodiments, the brakinginformation can be received from any braking system. In someembodiments, the hand contact and braking information can be received atstep 15402.

At step 15408, it is determined If there is hand contact with thesteering wheel. For example, it is determined if one or both hands arein contact with the touch steering wheel 134 based on the hand contactinformation. If the determination at step 15408 is NO (e.g., no handcontact with the touch steering wheel 134), the method returns to step15402. Otherwise, the method proceeds to step 15410. At step 15410, itis determined if the accelerator pedal of the motor vehicle 100 isengaged (e.g., depressed) or the brake pedal of the motor vehicle 100 isengaged (e.g., depressed) based on the braking information. If thedetermination at step 15410 is NO, the method returns to step 15402.Otherwise, the method proceeds to step 15412. At step 15412, the ECU12902 disengages (e.g., releases) the electric parking brake system 210.

Referring now to FIGS. 155A and 155B methods for controlling vehiclesystems based in part on hand contact transitions will be described.Specifically, FIG. 155A illustrates a method for controlling vehiclesystems based on hand contact transitions according to one embodiment.At step 15502, the ECU 12902 receives hand contact information (e.g.,monitoring information). The hand contact information can be receivedfrom the touch steering wheel system 134 and/or the EPS system 132. Atstep 15504, the ECU 12902 determines if a hand contact transition withthe steering wheel has occurred. For example, based on the hand contactinformation, it is determined if the number of hands in contact with thesteering wheel 134 has changed. More specifically, in the embodimentshown in FIG. 155A, it is determined if a transition has occurred fromone hand in contact with the touch steering wheel 134 to two hands incontact with the touch steering wheel 134. Alternatively, it can bedetermined if a transition from two hands in contact with the touchsteering wheel 134 to one hand in contact with the touch steering wheel134 has occurred. In some embodiments, at step 15504, the ECU 12902 candetermine if the transition has occurred within a predetermined periodof time.

If a hand contact transition is not detected at step 15504, the methodreturns to step 15502. Otherwise, the method proceeds to step 15506,where the ECU 12902 determines a driver state and/or driver state index.The driver state and/or driver state index is based on the hand contacttransition detected at step 15504. For example, a transition from onehand in contact with the steering wheel 134 to two hands in contact withthe steering wheel 134 can indicate the driver state is attentive andthe driver may be initiating a maneuver of the motor vehicle 100. Insome embodiments, the indication that the driver is initiating amaneuver with the motor vehicle 100 can be confirmed with steeringinformation as will be described with FIG. 155B. In another example, atransition from two hands in contact with the steering wheel 134 to onehand in contact with the steering wheel 134 can indicate the driverstate is distracted. In some embodiments, although a transition from twohands in contact with the steering wheel 134 to one hand in contact withthe steering wheel 134 has occurred, current steering information can becompared to stored steering information to determine the driver state asdescribed with FIG. 156. It is understood that in some embodiments, step15506 also includes determining if the driver state is attentive (e.g.,alert) or distracted.

At step 15508, the method includes modifying control of one or morevehicle systems based on the driver state. For example, if the driverstate is determined to be attentive, the ECU 12902 can control the lanedeparture warning system 222 and/or the blind spot indicator system 224by disabling these systems and/or setting the control type (e.g., systemstatus) of these systems to no control (e.g., OFF). Accordingly,warnings emitted by the lane departure warning system 222 and/or theblind spot indicator system 224 are deactivated and/or suppressed. Inanother embodiment, if the driver state is determined to be attentive,the ECU 12902 can control the lane keep assist system 226 by disablingthe system and/or setting the control type (e.g., system status) of thissystem to no control (e.g., OFF). In a further embodiment, themodification of the vehicle systems at step 15508 can be modified to theoriginal control type (e.g., system status) after a period of timeand/or after another hand contact transition is detected.

FIG. 155B illustrates a specific implementation of controlling a vehiclemode based in part on a hand contact transition. At step 15510, themethod includes the ECU 12902 receiving vehicle mode information from,for example, the vehicle mode selector system 238, and hand contactinformation, from, for example, the touch steering wheel system 134and/or the EPS system 132. At step 15512, the ECU 12902 determines if ahand contact transition with the steering wheel has occurred. Forexample, based on the hand contact information, it is determined if thenumber of hands in contact with the touch steering wheel 134 haschanged. More specifically, in the embodiment shown in FIG. 155B, it isdetermined if a transition has occurred from two hands in contact withthe touch steering wheel 134 to one hand in contact with the touchsteering wheel 134.

If the determination at step 15512 is NO, the method returns to step15510. If the determination at step 15512 is YES, the method proceeds tostep 15514 where the ECU 12902 determines a driver state and/or driverstate index. The driver state and/or driver state index is based on thehand contact transition detected at step 15512. At step 15516, themethod includes modifying the vehicle mode (e.g., switching the vehiclemode) based on the vehicle mode received at step 15510 and the handcontact transition. Thus, the ECU 12902 can control the vehicle modeselector system 238 to switch a mode at step 15516. In some embodiments,the vehicle mode is switched based on a look-up table 15518. Forexample, if the vehicle mode received at step 15502 is a sport mode, thevehicle mode is switched to comfort mode. If the vehicle mode receivedat step 15502 is a normal mode, the vehicle mode is switched to comfortmode. This modification allows for intuitive vehicle control based onthe driver state.

In some embodiments, it may not be safe to switch vehicle modes during adriving maneuver. Accordingly, in FIG. 155B, after a determination ofYES is made at step 15512, the method can optionally proceed to step15520, which includes receiving steering information. The steeringinformation can be analyzed to determine if the vehicle is currently ina maneuver and/or completing a maneuver. For example, a degree of yawrate, steering angle, and/or lateral G movement can be compared topredetermined thresholds to determine if the vehicle is currentlyperforming a maneuver (e.g., a turn, a sharp curve). Thus, at step15522, the method includes determining if a maneuver is in progress. If,the determination is NO, the method proceeds to step 15514. If thedetermination is YES, the method proceeds to step 15524 where it isdetermined if the maneuver is complete. If the maneuver is complete, themethod proceeds to step 15514. Otherwise, the method returns to step15520. Accordingly, the vehicle mode can be modified and/or switch at anappropriate time to ensure a safe and smooth transition.

Referring now to FIG. 156, a method for controlling a power steeringsystem of an electronic power steering system according to an exemplaryembodiment is shown. At step 15602, the method includes receivingsteering information, from, for example, the EPS system 132 and/or thetouch steering wheel system 134. At step 15604, the method includesdetermining a driver state and/or a driver state index based on thesteering information. In some embodiments, at step 15606, the driverstate index can be based on comparing the steering information receivedat step 15602 to stored steering information for an identified driver.For example, FIG. 24B illustrates an embodiment for controlling one ormore vehicle systems with identification of a driver.

Referring again to FIG. 156, at step 15608, the method includescontrolling the electronic power steering system 132 (e.g., a powersteering status) and the lane keep assist system 226 (e.g., a controltype and/or system status). More specifically, the power steering statusis set and the lane keep assist system 226 is enabled (e.g., turned ON).In some embodiments, a look-up table 15610 can be used to set the powersteering status. For example, if the driver state index is 1 or 2 (e.g.,driver is attentive/not drowsy), the power steering status can be set toauto and more steering assistance is provided to the driver according tothe lane keep assist system 226.

Referring now to FIG. 157, a method for controlling a low speed followsystem is shown. At step 15702, the method includes receivinginformation from a low speed follow system (e.g., monitoringinformation). For example, the ECU 12902 can receive information fromthe low speed follow system 212. At step 15704, the method can includedetermining a possible hazard based on the information from the lowspeed follow system. For example, the low speed follow system 212 canidentify a target vehicle in front of the motor vehicle 100 as apotential hazard. If a potential hazard is not detected at step 15704,the method can return to step 15702. Otherwise, the method proceeds tostep 15706.

At step 15706, the method includes receiving head movement information(e.g., head look), for example, from a head movement monitoring system334, and/or eye gaze information, for example from an eye/facialmovement monitoring system 332, and/or hand contact information from atouch steering wheel system 134. The head movement information caninclude information about a head pose and a head look of the driver asdiscussed above in Section III (B) (2) and with FIGS. 16A, 16B and 17.The hand contact information can include information about the contactand position of the driver's hands with respect to the touch steeringwheel as described with FIG. 18. In some embodiments, the head movementinformation, eye gaze information and/or the hand contact informationcan be received at step 15702.

At step 15708, the ECU 12902 can analyze hand contact information, theeye gaze information and/or the head movement information relative tothe information received from low speed follow system 212 (e.g.,relative to the potential hazard). Said differently, the ECU 12902 candetermine a trajectory and potential collision with a target vehicle, adirection of the head movement (e.g., a head look) and/or eye gazerelative to the target vehicle and hand contact with the steering wheel.Accordingly, at step 15710, the method includes determining a driverstate and/or driver state index. For example, the driver state is basedon the monitoring information (e.g., the low speed follow systeminformation, the hand contact information, the eye gaze information,and/or the head movement information) and the potential hazard. In someembodiments, step 15710 can also include determining if the driver isattentive and/or distracted based on the driver state and/or the driverstate index. More specifically, in FIG. 157, the ECU 12902 determinesthe driver state based on at least the hand contact information, eyegaze information and/or head movement information received at step 15706and the analysis of the hand contact information, eye gaze informationand/or head movement information relative to the potential hazard atstep 15708. Said differently, the driver state and/or the driver stateindex is based at least in part on the low speed follow systeminformation, the head movement information, the eye gaze informationand/or the hand contact information.

For example, if the head position and contact information indicates thedriver has at least one hand on the wheel and the head look is aforward-looking head look of the driver, the driver state is determinedto be attentive at step 15710. If the hand contact information indicatesthe driver has at least one hand on the wheel and the head look is anon-forward-looking head look of the driver, the driver state isdetermined to be distracted at step 15710. If the hand contactinformation indicates the driver has no hands on the wheel, the driverstate is determined to be distracted at step 15710.

At step 15712, the method includes controlling the low speed followsystem based on the driver state and/or driver state index. Morespecifically, the ECU 12902 sets the low speed follow system status(e.g., control status/type) based on the driver state. For example, ifthe driver state is distracted, the ECU 12902 can set the control typeof the low speed follow system 212 to standard control and modify thetouch steering wheel 134 (e.g., at step 15714) to provide visualwarnings (e.g., to put at least one hand on the wheel and/or lookforward) at step 15714. Accordingly, the visual warnings inform thedriver 102 of the driver state.

If the driver state is attentive, the ECU 12902 can set the control typeof the low speed follow system 212 to auto control. Accordingly, lowspeed follow system 212 in conjunction with the automatic cruise controlsystem 216 will move relative to the target vehicle. Thus, the ECU 12902can also control the automatic cruise control system 216 to slow downand/or increase a distance between the motor vehicle 100 and the targetvehicle. Further, the ECU 12902 can control a lane keep assist system226 (e.g., enable the lane keep assist system 226) based on the driverstate to help keep the vehicle within the current lane markers.

Referring now to FIGS. 158A and 158B, a schematic view of controlling alow speed follow system and a visual device (e.g., a visual device on asteering wheel) in accordance with the method of FIG. 157 is shown. InFIG. 158A, the motor vehicle 100 (e.g., host vehicle) is travellingbehind a preceding vehicle 15802 (e.g., target vehicle). The vehicle 100includes the automatic cruise control system 216 and the low speedfollow system 212 is set to a status of ON. Here, the head look of thedriver 102 is forward-looking relative to the motor vehicle 100 and onehand is in contact with the touch steering wheel 134. Accordingly, basedon the potential hazard with the target vehicle, the head movementinformation, and the hand contact information, the driver state isdetermined to be attentive. Accordingly, the ECU 12902 controls the lowspeed follow system 212 and/or the automatic cruise control system 216to maintain a predetermined headway distance 15804 behind the precedingvehicle 15802 (e.g., standard control, auto control). In a stop and gosituation, the motor vehicle 100 will move, without physical interaction(e.g., switching a button to engage the low speed follow system), inrelation to the preceding vehicle 15802 when the driver is attentive.

In FIG. 158B the motor vehicle 100 (e.g., host vehicle) is travellingbehind the preceding vehicle 15802 (e.g., target vehicle). The vehicle100 includes the automatic cruise control system 216 and the low speedfollow system 212 is set to a status of ON. Here, the head look of thedriver 102 is forward-looking, but the driver 102 does not have anyhands in contact with the touch steering wheel 134. According, based onthe potential hazard, the head movement, and the hand contact with thetouch steering wheel 134, the driver state is determined to bedistracted. Therefore, the ECU 12902 can control the low speed followsystem 212 by setting the system status to disabled and the ECU 12902can control visual devices 140 (e.g., the light bar on the touchsteering wheel 134) to provide warning signals 15806 to the driver 102.

When the driver 102 contacts the steering wheel with at least one hand,as shown in FIG. 158A, the motor vehicle 100 will move, in relation tothe preceding vehicle 15802 (e.g., the driver state is determined to beattentive based on a forward head look of the driver and at least onehand in contact with the touch steering wheel system 134). Thisillustrative example shows how operation (e.g., ON, OFF) of a vehiclesystem can change within milliseconds based on the driver state.

FIG. 159 illustrates an alternative embodiment of the process of FIG.157. At step 15902, the method includes receiving low speed followinformation (e.g., monitoring information), from, for example, the lowspeed follow system 212. At step 15904, it is determined if there is apotential hazard based on the information received at step 15902, forexample, a potential hazard with a preceding vehicle. If thedetermination at step 15904 is NO, the method returns to step 15902. Ifthe determination at step 15904 is YES, the method proceeds to step15906. At step 15906, the method includes receiving hand contactinformation, from, for example, the EPS system 132, and/or the touchsteering wheel system 134. In some embodiments, the hand contactinformation can be received at step 15902.

At step 15908, the ECU 12902 determines if there is hand contact withthe steering wheel. More specifically, it is determined if at least onehand is in contact with the steering wheel based on the informationreceived at step 15904. If NO, at step 15908, the ECU 12902 sets thesystem status of the low speed follow system 212 to manual control.Accordingly, the low speed follow system 212 will not be activatedwithout a manual input from the driver. Further, similar to the methodof FIG. 157, a visual indicator can be activated based on the status ofthe low speed follow system 212 and the driver state (e.g., the handcontact determination at step 15908). For example, a light bar of thetouch steering wheel 134 (See. FIG. 18) can be activated to emit a redcolor thereby indicating to the driver that the low speed follow systemis in a manual (e.g., not standard) state.

If it is determined that at least one hand is on the steering wheel, atstep 15908, the method includes receiving head movement and/or eye gazeinformation at step 15912 from the head movement monitoring system 334and/or eye/facial movement monitoring system 332. The head movementinformation can include information about a head pose and a head look ofthe driver as discussed above in Section III (B) (2) and with FIGS. 16A,16B and 17. It is understood that the information from the head movementand/or eye gaze information from the head movement monitoring system 334and/or eye/facial movement monitoring system 332 can be received at step15912. At step 15914, it is determined if the head look and/or eye gazeis forward-looking based on the head movement information and/or eyegaze information.

If the head look and/or eye gaze is not forward-looking, the methodproceeds to step 15910. If the head look and/or eye gaze isforward-looking at step 15914, then at step 15916, the method includesthe ECU 12902 setting the low speed follow system 212 status to autocontrol (e.g., turned ON, standard control). Accordingly, the low speedfollow system 212 will be activated and move automatically based on thepreceding vehicle. For example, in a stop and go situation, if the motorvehicle 100 is stopped and the preceding vehicle is stopped, the hostvehicle will automatically move according to the preceding vehicle whenthe preceding vehicle moves without manual input from the driver.Further, a visual indicator can be activated based on the status of thelow speed follow system and the driver state (e.g., hand contact, eyegaze and/or head look). For example, a light bar of the touch steeringwheel 134 (See FIG. 18) can be activated to emit a green color therebyindicating to the driver that the low speed follow system 212 is in anauto state.

Referring now to FIG. 160, a method for operating an automatic cruisecontrol system in response to a driver state is illustrated. At step16002, the method includes the ECU 12902 receiving information from anautomatic cruise control system 216 (e.g. monitoring information). Atstep 16004, the ECU 12902 determines if a potential hazard exists basedon the information from the automatic cruise control system 216. Forexample, the ECU 12902 can detect a potential hazard as an object (e.g.,a target vehicle) in front of the motor vehicle 100. If a potentialhazard does not exist, the method returns to step 16002. Otherwise, themethod proceeds to step 16006.

At step 16006, the method includes receiving head movement information(e.g., head look), for example from a head movement monitoring system334 and/or eye gaze information, for example from an eye/facial movementmonitoring system, and hand contact information from a touch steeringwheel system 134. The head movement information can include informationabout a head pose and a head look of the driver as discussed above inSection III (B) (2) and with FIGS. 16A, 16B, and 17. The hand contactinformation can include information about the contact and position ofthe driver's hands with respect to the steering wheel as described withFIG. 18. It is understood that the head movement information, eye gazeinformation and the hand contact information can be received at step16002. It should be noted that steps 16002 and 16004 are optional. Inother words, the method can begin at step 16006 with receiving handcontact, eye gaze and/or head movement information as discussed below.

At step 16008, the ECU 12902 can analyze the hand contact information,eye gaze information and/or the head movement information relative tothe information received from the automatic cruise control system 216(e.g., relative to the potential hazard). Said differently, the ECU12902 can determine a head movement (e.g., head look) and/or eye gazerelative to the potential hazard, for example, the target vehicle, andhand contact relative to the touch steering wheel 134. Accordingly, atstep 16010, the method includes determining a driver state and/or driverstate index. For example, the driver state is determined based onmonitoring information relative to the potential hazard. Morespecifically, in FIG. 160, the ECU 12902 determines the driver statebased on at least the head movement information and/or eye gazeinformation, hand contact information received at step 16006, and theanalysis of the head movement information and/or eye gaze information,and hand contact information relative to the target vehicle at step16008. Said differently, the driver state and/or the driver state indexis based at least in part on the head movement information and/or theeye gaze information, the hand contact information, and the targetvehicle (e.g., the potential hazard).

In some embodiments, step 16010 can also include determining if thedriver is attentive and/or distracted based on the driver state and/ordriver state index. For example, if the hand contact informationindicates the driver 102 has at least one hand on the touch steeringwheel 134 and the head look and/or eye gaze is a forward-looking headlook and/or eye gaze of the drive 102, the driver state is determined tobe attentive at step 16010. If the hand contact information indicatesthe driver 102 has at least one hand on the touch steering wheel 134 andthe head look and/or eye gaze is a non-forward-looking head look and/oreye gaze of the driver 102, the driver state is determined to bedistracted at step 16010.

At step 16012, the ECU 12902 modifies one or more vehicle systems basedon the driver state. In one embodiment, the ECU 12902 modifies a controltype of one or more vehicle systems including, for example, the lanekeep assist system 226 and the automatic cruise control system 216. Forexample, if the driver state is distracted, the ECU 12902 can set thecontrol type (e.g., system status) of the automatic cruise controlsystem 216 to partial and/or full control (e.g., ON). Thus, the ECU12902 can control the automatic cruise control system 216 to slow downand/or increase a space between the motor vehicle 100 and the targetvehicle automatically. Further, if the driver state is distracted, theECU 12902 can set the control type of the lane keep assist system 226 topartial and/or full control (e.g., ON). Thus, the lane keep assistsystem 226 can provide assistance to keep the motor vehicle 100 withinthe current lane markers. In this way, the vehicle 100 can continue todrive within the current lane at its set cruise speed without requiringthe driver 102 to be actively driving the vehicle (e.g., hands on thewheel, foot on the accelerator pedal, etc., while still requiring thedriver 102 to be monitoring the progress of the vehicle (e.g. lookingforward)).

Referring now to FIGS. 161A and 161B, a schematic view of controllingone or more vehicle systems in accordance with the method of FIG. 160 isshown. In FIG. 161A, the motor vehicle 100 is travelling behind apreceding vehicle 16102 with automatic cruise control system 216 systemstatus set to ON. Here, the head look of the driver 102 isforward-looking relative to the motor vehicle 100 and one hand is incontact with the touch steering wheel 134. Based on the target vehicle,the head movement information and/or eye gaze information and the handcontact information, the ECU 12902 determines the driver state asattentive and the ECU 12902 sets the automatic cruise control system 216to a medium gap. Therefore, the motor vehicle 100 maintains apredetermined headway distance 16104 behind the preceding vehicle 16102.

In FIG. 161B, the head look of the driver 102 is not forward-lookingrelative to the motor vehicle 100 (e.g., head looking down) and one handis in contact with the touch steering wheel 134. Based on the targetvehicle, the head movement information and/or the eye gaze information,and the hand contact information, the ECU 12902 determines the driverstate as distracted and the ECU 12902 sets the automatic cruise controlsystem 216 to a maximum gap. Accordingly, the motor vehicle 100 controlsthe operation of the automatic cruise control system 216 so that theautomatic cruise control system 216 increases the headway distance to asecond headway distance 16106. In another embodiment, the ECU 12902 setsthe automatic cruise control system 216 to manual therefore requiringthe driver the manually set control parameters of the automatic cruisecontrol system 216.

In FIG. 161C, the motor vehicle 100 is travelling behind a precedingvehicle 16102 with automatic cruise control system 216 system status setto ON. Here, the head look of the driver 102 is forward-looking relativeto the motor vehicle 100 and two hands are in contact with the touchsteering wheel 134. Based on the target vehicle, the head movementinformation and/or the eye gaze information, and the hand contactinformation, the ECU 12902 determines the driver state as attentive andthe ECU 12902 sets the automatic cruise control system 216 to a minimumgap. Accordingly, the motor vehicle 100 controls the operation of theautomatic cruise control system 216 so that the automatic cruise controlsystem 216 decreases the headway distance to a third headway distance16108. As can be seen, since the driver 102 in FIG. 161C has both handson the touch steering wheel 134, the third headway distance (e.g.,minimum gap) is smaller than the headway distance 16104 in FIG. 161Awhere the drier 102 only has one had on the touch steering wheel 134.

FIG. 162 illustrates a method for controlling an automatic cruisecontrol system and a lane keep assist system according to anotherexemplary embodiment. At step 16202, the method includes receivingautomatic cruise control information (e.g., monitoring information)from, for example, the automatic cruise control system 216. At step16204, it is determined if a potential hazard exists. For example, it isdetermined if a potential hazard exists with a preceding vehicle basedon the information from the automatic cruise control system 216. If thedetermination at step 16204 is NO, the method returns to step 16202. Ifthe determination at step 16204 is yes, the method proceeds to step16206. It should be noted that steps 16202 and 16204 are optional. Inother words, the method can begin at step 16206 with receiving handcontact and head movement information as discussed below.

At step 16206, the method includes the ECU 12902 receiving hand contactinformation from the touch steering wheel system 134 and head movementinformation from the head movement monitoring system 334 and/or eye gazeinformation from the eye/facial movement monitoring system 332. In someembodiments, the hand contact information, the head movement informationand/or the eye gaze information can be received at step 16202. The headmovement information can include information about a head pose and ahead look of the driver as discussed above in Section III (B) (2) andwith FIGS. 16A, 16B and 17.

At step 16208, the method includes determining if there is hand contact(e.g., at least one hand) with the steering wheel based on the handcontact information. More specifically, in the embodiment shown in FIG.162 it is determined if both hands are off the touch steering wheel 134.If at least one hand is detected on the touch steering wheel 134, themethod proceeds to step 16214. Otherwise, the method proceeds to step16210. At step 16210, the method includes the ECU 12902 setting the lanekeep assist system 226 status to auto control. Further, at step 16212,the method includes setting the automatic cruise control system 216status based on at least one of head look and head look duration (e.g.,based on the head movement information). For example, if the head lookis forward-looking, the headway distance of the automatic cruise controlsystem 216 is set to a minimum gap. If the head look is in anon-forward-looking direction with a duration of more than apredetermined number of seconds (e.g., 2 seconds), the headway distanceof the automatic cruise control system 216 is set to a medium gap. Ifthe head look is in any direction with a duration of less than apredetermined number of seconds (e.g., 2 seconds), the headway distanceof the automatic cruise control system 216 is set to a minimum gap.

Returning to step 16208, if there at least one hand on the steeringwheel, at step 16214, the ECU 12902 sets the ECU 12902 sets theautomatic cruise control system 216 to manual control (e.g., the headwaydistance is set by manual input). At step 16216, the method includessetting a status of the lane keep assist system 226 based on at leastone of hand contact, head look and head look duration (e.g., based onthe head movement information). For example, if the left hand or righthand is detected on the wheel and the head look is forward-looking, thelane keep assist system 226 status is set to standard control. If theleft hand or right hand is detected on the wheel and the head look is ina non-forward-looking direction for more than predetermined amount oftime (e.g., 2 seconds), the lane keep assist system 226 status is set toauto control. In this way, the vehicle 100 can continue to drive withinthe current lane at its set cruise speed without requiring the driver102 to be actively driving the vehicle (e.g., hands on the wheel, footon the accelerator pedal, etc., while still requiring the driver 102 tobe monitoring the progress of the vehicle (e.g. looking forward)).

As discussed briefly above, the lane keep assist system 226 in an autocontrol status can automatically control the electronic power steeringsystem 132 to keep the vehicle in a predetermined lane based onidentifying and monitoring lane markers of the predetermined lane. Insome embodiments, there may be a break in the lane markers and/or thelane markers may not be identifiable. Accordingly, the controlparameters of the lane keep assist system 226 can be modified based on adriver state in an auto control mode. Referring now to FIG. 163 a methodfor controlling an automatic cruise control system and a lane keepassist system is shown. At step 16302, the method includes receivinglane keep assist system and/or navigation information (e.g., monitoringinformation). At step 16304, the method includes determining if there isa break in lane markers adjacent to the vehicle based on the monitoringinformation received at step 16302. If the determination at 16304 is NO,the method returns to step 16302. Otherwise, the method proceeds to step16306. At step 16306, the method includes receiving blind spot indicatorsystem information from the blind spot indicator system 224 and headmovement information from the head movement monitoring system 334. It isunderstood that the blind spot indicator information and the headmovement information can be received at step 16302. It should beunderstood that eye gaze information may be used instead of or inaddition to head movement information to determine where the driver 102is looking. At step 16308, it is determined if there is a potentialhazard (e.g., a target vehicle in a blind spot monitoring zone) relativeto the break. If the determination at step 16308 is YES, the methodreturns to step 16302. Otherwise, the method proceeds to step 16310where the cruise control system 216 and the lane keep assist system 226are modified based on head movement, the current lane and the break inthe lane.

FIG. 164 illustrates a more detailed example of the method of FIG. 163.At step 16402, the method includes receiving lane keep assist systeminformation and/or navigation information, for example from the lanekeep assist system 226 and/or the navigation system 230 (e.g.,monitoring information). At step 16404, it is determined if there is abreak in the adjacent lane markers. The break can be identified, forexample, by optical sensors of the lane keep assist system 226 and/orinformation from the navigation system 230. For example, a break may beidentified if the adjacent (e.g., adjacent to the motor vehicle 100)lane markers are not identifiable by the lane keep assist system (e.g.,the lane markers are not clear, are obstructed, have faded away). Inanother embodiment, a break may occur based on current traffic patterns,for example, an exit off a highway. If the determination at step 16404is NO, the method proceeds back to step 16402. If the determination atstep 16404 is YES, the method proceeds to step 16406. At step 16406, themethod includes receiving head movement information. In someembodiments, the head movement information can be received by the headmonitoring system 334 and the head movement information can be receivedat step 16402. It should be understood that eye gaze information may beused instead of or in addition to head movement information to determinewhere the driver 102 is looking.

At step 16408 it is determined if the head look is forward-looking basedon the head movement information. If YES, at step 16410, the methodincludes the ECU 12902 controlling the automatic cruise control system216 and the lane keep assist system 226 to maintain the motor vehiclesystem in the current lane according to the head look for the side ofthe vehicle without a break in the adjacent lane. Thus, the ECU 12902can set the control status of the automatic cruise control system 216and the lane keep assist system 226 to auto control, and the lane keepassist system 226 will maintain the vehicle in the current lane based onthe adjacent lane marker without the break.

If NO, at step 16408, the method includes determining if the head lookis directed towards the break in the adjacent lanes at step 16412. IfNO, at step 16414, the method proceeds to step 16410. If YES, at step16412, the method includes receiving information from a blind spotindicator system 224 at step 16414. At step 16416, it is determined if apotential hazard exists relative to the break in the adjacent lanesbased on the information received at step 16414. For example, apotential hazard relative to the break exists if there is a targetvehicle in a blind spot monitoring zone of the motor vehicle 100 in thesame direction of the break in the adjacent lanes.

If YES, at step 16416, the ECU 12902 modifies the automatic cruisecontrol system 216 and lane keep assist system 226 according to the headlook and the break in the adjacent lane at step 16418. Accordingly, thelane keep assist system 226 can allow the vehicle to move according tothe head look of the driver and break in the adjacent lanes. If NO, atstep 16416, the method proceeds to 16410 for maintaining the vehicle inthe current lane via the automatic cruise control system 216 and lanekeep assist system 226 in the current lane based on lane markerinformation (e.g., from the lane keep assist system) for the side of thevehicle without a break in the adjacent lane. It is appreciated that avisual indicator can also be provided to the driver based on the driverstate and the system control of the vehicle.

Referring now to FIGS. 165A and 165B, an illustrative example accordingto the method of FIG. 164 is shown. Here, the motor vehicle 100 istravelling in a current lane 16502 with an adjacent left lane marker16504 and an adjacent right lane marker 16506. As the motor vehicle 100approaches a break 16508 in the adjacent right lane marker 16506, theECU 12902 can determine a head look of the driver based on head movementinformation. In FIG. 165A, the driver 102 has a head look directedforward (e.g., not towards the break 16508). Accordingly, the ECU 12902controls the automatic cruise control system 216 and lane keep assistsystem 226 to maintain the motor vehicle 100 position in the currentlane 16502. Thus, the lane keep assist system 226 will use the adjacentleft lane marker 16504 (e.g., the adjacent lane without the break) toguide the motor vehicle 100.

In FIG. 165B, the driver 102 has a head look directed towards the break16508. Additionally, a target vehicle 16510 is a predetermined distance16512 forward of the motor vehicle 100. If the target vehicle 16510 doesnot present a hazard, the ECU 12902 controls the automatic cruisecontrol system 216 and lane keep assist system 226 based on the headlook of the driver and the break 16508, thereby controlling the vehicleto turn right.

While various embodiments have been described, the description isintended to be exemplary, rather than limiting and it will be apparentto those of ordinary skill in the art that many more embodiments andimplementations are possible that are within the scope of theembodiments. Accordingly, the embodiments are not to be restrictedexcept in light of the attached claims and their equivalents. Inaddition, various modifications and changes can be made within the scopeof the attached claims.

In accordance with one aspect, a method of controlling vehicle systemsin a motor vehicle includes receiving monitoring information from one ormore monitoring systems and determining a plurality of driver statesbased on the monitoring information from the one or more monitoringsystems. The method also includes determining a combined driver stateindex based on the plurality of driver states and modifying control ofone or more vehicle systems based on the combined driver state index.

Determining the combined driver state index is based on at least oneselected of the plurality of driver states, at least one differentselected of the plurality of driver states, and at least one otherdifferent selected of the plurality of driver states. Further,determining the combined driver state index is based on at least a firstdriver state selected from the plurality of driver states, a seconddriver state selected from the plurality of driver states, and a thirddriver state selected from the plurality of driver states.

Determining the combined driver state index includes aggregating the atleast one selected of the plurality of driver states, the at least onedifferent selected of the plurality of driver states, and the at leastone other different selected of the plurality of driver states. Inanother embodiment, determining the combined driver state index includesaggregating the first driver state selected from the plurality of driverstates, the second driver state selected from the plurality of driverstates and the third driver state selected from the plurality of driverstates. In a further embodiment, determining the combined driver stateindex includes determining an average of the at least one selected ofthe plurality of driver states, the at least one different selected ofthe plurality of driver states, and the at least one other differentselected of the plurality of driver states. In an additional embodiment,determining the combined driver state index includes determining anaverage of the first driver state selected from the plurality of driverstates, the second driver state selected from the plurality of driverstates, and the third driver state selected from the plurality of driverstates.

The plurality of driver states being at least one of the followingdriver state types: a physiological driver state, a behavioral driverstate, or a vehicular-sensed driver state. The plurality of driverstates are based on at least one of physiological information,behavioral information, and vehicular-sensed information. Morespecifically, the physiological driver state is based on physiologicalinformation, the behavioral driver state is based on behavioralinformation, and the vehicular-sensed driver state is based on vehicleinformation.

In one embodiment, the at least one selected of the plurality of driverstates is a physiological driver state, the at least one differentselected of the plurality of driver states is a behavioral driver state,and the at least one other different selected of the plurality of driverstates is a vehicular-sensed driver state. Further, the at least oneselected of the plurality of driver states is based on physiologicalinformation, the at least one different selected of the plurality ofdriver states is based on behavioral information, and the at least oneother different selected of the plurality of driver states is based onvehicular-sensed information. The physiological information, thebehavioral information and the vehicular-sensed information are types ofmonitoring information received from one or more monitoring systems.

In one embodiment, the first driver state selected from the plurality ofdriver states is a physiological driver state, the second driver stateselected from the plurality of driver states is a behavioral driverstate, and the third driver state selected from the plurality of driverstates is a vehicular-sensed driver state. In another embodiment, thefirst driver state selected from the plurality of driver states is abased on physiological information, the second driver state selectedfrom the plurality of driver states is based on behavioral information,and the third driver state selected from the plurality of driver statesis based on vehicular-sensed information.

In a further embodiment, at least one selected of the plurality ofdriver states is a physiological driver state and the at least onedifferent selected of the plurality of driver states is a behavioraldriver state. The physiological driver state and the behavioral driverstate are based on information from one of the monitoring systems. Theone of the monitoring systems includes a sensor for receivingphysiological information and behavioral information. The physiologicaldriver state is based on the physiological information and thebehavioral driver state is based on the behavioral information. In oneembodiment, the physiological information is heart rate information andthe behavioral information is head movement information. Further, thesensor is an optical sensor for receiving the physiological informationand the behavioral information.

Determining the combined driver state also includes determining if thecombined driver state indicates a distracted driver state. Morespecifically, determining the combined driver state includes determiningif the first driver state selected from the plurality of driver statesindicates a distracted driver state and the second driver state selectedfrom the plurality of driver states indicates a distracted driver state.

Upon determining at least one of the first driver state selected fromthe plurality of driver states state or the second driver state selectedfrom the plurality of driver states indicates a distracted driver state,the combined driver state is determined to indicate a distracted driverstate. Upon determining at least one of the first driver state selectedfrom the plurality of driver states or the second driver state selectedfrom the plurality of driver states indicates a non-distracted driverstate, the combined driver state is determined to indicate anon-distracted driver state. Further, upon determining the third driverstate selected from the plurality of driver states indicates adistracted driver state, the combined driver state is determined toindicate a distracted driver state.

In one embodiment, determining the combined driver state is based on atleast two selected of the plurality of driver states. The at least twoselected of the plurality of driver states being the same driver statetype. For example, the at least one selected of the plurality of driverstates and the at least one different selected of the plurality ofdriver states are the same driver state type. As another example, thefirst driver state selected from the plurality of driver states and thesecond driver state selected from the plurality of driver states are thesame driver state type. Further, the third driver state selected fromthe plurality of driver states is a different driver state than thefirst driver state and the second driver state. Accordingly, in oneembodiment, the first driver state selected from the plurality of driverstates and the second driver state selected from the plurality of driverstates are behavioral driver states and the third driver state is aphysiological driver state or a vehicular-sensed driver state.

In accordance with another embodiment, a method of controlling vehiclesystems in a motor vehicle includes receiving monitoring informationfrom one or more monitoring systems and determining a plurality ofdriver states based on the monitoring information from the one or moremonitoring systems. The plurality of driver states being at least one ofthe following types of driver states: physiological driver state,behavioral driver state, and vehicular-sensed driver state. The methodalso includes determining a combined driver state index based on theplurality of driver states and modifying control of one or more vehiclesystems based on the combined driver state index. Determining thecombined driver state index is based on at least a first driver stateselected from the plurality of driver states, a second driver stateselected from the plurality of driver states and a third driver stateselected from the plurality of driver states. The first driver state,the second driver state, and the third driver state are each a differenttype of driver state. In one embodiment, the first driver state and thesecond driver state are the same type of driver state and the thirddriver state is a different type of driver state than the first driverstate and the second driver state.

Further, determining the combined driver state index includes comparingthe one or more of the plurality of driver states to at least onethreshold and includes comparing at least one of the first driver state,the second driver state and the third driver state to respectivethresholds, and determining the combined driver state index based on thecomparison. In one embodiment, determining the combined driver stateindex further includes comparing the first driver state to a firstdriver state threshold, comparing the second driver state to a seconddriver state threshold and comparing the third driver state to a thirddriver state threshold, and determining the combined driver state basedon the comparison. Upon determining the first driver state meets thefirst driver state threshold and the second driver state meets thesecond driver state threshold, the combined driver state index is basedon the first driver state and the second driver state.

Further, determining the combined driver state index includes confirmingat least one selected of the plurality of driver states with at leastone different selected of the plurality of driver states, and confirmingthe at least one selected of the plurality of driver states, the atleast one different selected of the plurality of driver states, with atleast another one of the plurality of driver states. Confirming includesdetermining if the at least one selected of the plurality of driverstates and the at least one different selected of the plurality ofdriver states indicate a distracted driver state. Upon determining theat least one selected of the plurality of driver states and the at leastone different selected of the plurality of driver states indicate adistracted driver state, determining the combined driver state index isbased on the at least one selected of the plurality of driver states andthe at least one different selected of the plurality of driver states.

In one embodiment, confirming the at least one selected of the pluralityof driver states with the at least one different selected of theplurality of driver states further includes comparing the at least oneselected of the plurality of driver states to a first threshold andcomparing the at least one different selected of the plurality of driverstates to a second threshold. The first threshold and the secondthreshold indicate a distracted driver state. Upon determining the atleast one selected of the plurality of driver states meets the firstthreshold and the at least one different selected of the plurality ofdriver states meets the second threshold, determining the combineddriver state index is based on the at least one selected of theplurality of driver states and the at least one different selected ofthe plurality of driver states. The first driver state threshold, thesecond driver state threshold and the third driver state threshold arevalues that indicate a distracted driver state. In one embodiment, thefirst driver state threshold, the second driver state threshold and thethird driver state threshold are pre-determined thresholds based on atleast one of: the type of driver state, the monitoring information usedto determine the plurality of driver states, and an identity of thedriver.

In one embodiment, the method includes modifying the first driver statethreshold, the second driver state threshold and the third driver statethreshold based on at least one of: the type of driver state, themonitoring information used to determine the plurality of driver states,and the identity of the driver. The threshold, the first driver statethreshold, the second driver state threshold and the third driver statethreshold are determine and/or modified based on the identity of thedriver, the identity of the driver determined by one of the monitoringsystems. In another embodiment, the threshold, the first driver statethreshold, the second driver state threshold and the third driver statethreshold is determine and/or modified based on learned baseline dataassociated with the driver. In a further embodiment, the threshold, thefirst driver state threshold, the second driver state threshold and thethird driver state threshold are determine and/or modified based onnormative data for other drivers with similar characteristics of thedriver. In still another embodiment, the threshold, the first driverstate threshold, the second driver state threshold and the third driverstate threshold is determine and/or modified based on a pattern ofmonitoring information over a period of time associated with the driver.In some embodiments, the first driver state threshold, the second driverstate threshold and the third driver state threshold is determinedand/or modified based on monitoring information indicating aninattentive driver.

In one embodiment, the first driver state is a vehicular-sensed driverstate based on steering wheel monitoring information and the firstdriver state threshold is a number of steering wheel jerks over theperiod of time that indicates the driver is distracted. In anotherembodiment, the first driver state is a behavioral driver state based onhead movement monitoring information and the first driver statethreshold is a number of head nods based on the head movement monitoringinformation over the period of time that indicates the driver isdistracted.

In accordance with a further embodiment, a method of controlling vehiclesystems in a motor vehicle includes receiving monitoring informationfrom a plurality of monitoring systems and determining a plurality ofdriver states based on the monitoring information from the plurality ofmonitoring systems. The method also includes determining a combineddriver state index based on the plurality of driver states and modifyingcontrol of one or more vehicle systems based on the combined driverstate index. The method further includes determining a potential hazardbased on monitoring information from one or more vehicle systems.Additionally, the method includes determining if the driver isdistracted based on the combined driver state index. The method alsoincludes determining an auto control status of the vehicle or one ormore vehicle systems.

Upon determining the driver is not distracted, modifying control of oneor more vehicle systems includes changing a control status of one ormore vehicle systems to no control. Upon determining the driver isdistracted and the auto control status is set to auto, modifying controlof one or more vehicle systems includes changing a control status of oneor more vehicle systems to auto control.

Determining the combined driver state index is based on analyzing headmovement information and hand contact information relative to thepotential hazard. The head movement information and the hand contactinformation are received from the plurality of monitoring systems.

In one embodiment, upon determining the potential hazard is a lanedeviation based on monitoring information from a lane departure warningsystem, determining the combined driver state index includes analyzinghead movement information relative to the lane deviation. In anotherembodiment, upon determining the potential hazard is a target vehicle ina blind spot monitoring zone of the vehicle based on monitoringinformation from a blind spot indicator system, determining the combineddriver state index includes analyzing head movement information or handcontact information relative to the target vehicle or the blind spotmonitoring zone.

In another embodiment, upon determining the potential hazard is apreceding vehicle in front of the vehicle based on monitoringinformation from an automatic cruise control system, determining thecombined driver state index includes analyzing head movement informationor hand contact information relative to the preceding vehicle. Analyzinghead movement information includes determining a head look direction inrelation to a direction of the hazard. Analyzing hand contactinformation includes determining contact of a least one hand of a driverwith a steering wheel of the vehicle. Upon determining the head lookdirection is forward-looking relative to the vehicle or the head lookdirection is directed in the same direction as the direction of thehazard, the combined driver state index is determined to be attentive,and modifying control of the one or more vehicle systems includessetting a control status of the one or more vehicle systems to nocontrol. Upon determining the head look direction is forward-lookingrelative to the vehicle or the head look direction is directed in thesame direction as the direction of the hazard and upon determining atleast one hand of the driver is in contact with the steering wheel, thecombined driver state index is determined to be attentive, and modifyingcontrol of the one or more vehicle systems includes setting a controlstatus of the one or more vehicle systems to no control.

In another embodiment, upon determining the potential hazard is apreceding vehicle based on monitoring information from a low speedfollow system and the auto control mode is set to ON, determining thecombined driver state index includes analyzing head movement informationor hand contact information. Upon determining at least one hand of adriver is in contact with a steering wheel of the vehicle and adirection of a head look of the driver based on the head monitoringinformation is in a forward-looking direction relative to the vehicle, acontrol status of the low speed follow system is set to auto control.Further, upon determining no hand contract with a steering wheel of thevehicle, a control status of a lane keep assist system is set to autocontrol and a control status of an automatic cruise control status isset based on the head monitoring information. The head monitoringinformation includes a head look direction and a head look duration. Ina further embodiment, upon determining at least one hand is in contactwith a steering wheel of the vehicle, a control status of an automaticcruise control system is set to manual control and a control status of alane keep assist system is set based on the head monitoring information.The head monitoring information includes a head look direction and ahead look duration.

What is claimed is:
 1. A method of controlling a vehicle, comprising:receiving vehicle information from one or more vehicle systems;determining a vehicular state of the vehicle based on the vehicleinformation, wherein the vehicular state indicates the vehicle is in astopped state; receiving monitoring information about a driver from amonitoring system, wherein the monitoring information includes headmovement information about the driver; determining a driver state indexbased on the head movement information by analyzing a head-look of thedriver relative to the vehicle; and modifying control of the vehicle toautomatically move based on the vehicular state and the driver stateindex.
 2. The method of claim 1, wherein if the head-look of the driveris forward-looking relative to the vehicle, and the driver is determinedto be attentive, modifying control of the vehicle includes controllingthe vehicle to automatically move forward with respect to a precedingvehicle.
 3. The method of claim 1, wherein the monitoring informationincludes hand contact information about contact between the driver and asteering wheel of the vehicle.
 4. The method of claim 3, whereinmodifying control of the vehicle includes modifying control of thevehicle to automatically move based on the vehicular state, the driverstate index, and the hand contact information.
 5. The method of claim 3,wherein modifying control of the vehicle includes modifying control ofthe vehicle to automatically move without manual input from the driverbased on the vehicular state, the driver state index, and the handcontact information.
 6. The method of claim 3, wherein if the head-lookof the driver is forward-looking relative to the vehicle based on thehead movement information, and there is no hand contact with thesteering wheel based on the hand contact information, modifying controlof the vehicle includes controlling the vehicle to automatically moveforward.
 7. The method of claim 3, wherein determining the driver stateindex includes determining a plurality of driver states based on themonitoring information, wherein the plurality of driver states includesa first driver state comprising the head movement information and asecond driver state comprising the hand contact information.
 8. Themethod of claim 7, including determining a combined driver state indexbased on the plurality of driver states, and wherein modifying controlof the vehicle includes modifying control of the vehicle based on thecombined driver state index.
 9. The method of claim 1, wherein upondetermining the vehicle is in the stopped state based on the vehicularstate and the driver is attentive based on the driver state index,modifying control of the vehicle includes modifying a control status ofthe one or more vehicle systems to auto control without a manual inputfrom the driver.
 10. The method of claim 4, wherein the one or morevehicle systems includes a low speed follow system and an automaticcruise control system.
 11. A system for controlling a vehicle,comprising: a monitoring system; and an electronic control unitincluding a processor, wherein the processor: receives vehicleinformation from one or more vehicle systems; determines a vehicularstate of the vehicle based on the vehicle information, wherein thevehicular state indicates the vehicle is in a stopped state; receivesmonitoring information about a driver from the monitoring system,wherein the monitoring information includes head movement informationabout the driver; determines a driver state index based on the headmovement information by analyzing a head-look of the driver relative tothe vehicle; and modifies control of the vehicle to automatically movebased on the vehicular state and the driver state index.
 12. The systemof claim 11, wherein if the head-look of the driver is forward-lookingrelative to the vehicle, and the driver state is determined to beattentive, the processor modifies control of the vehicle toautomatically move forward with respect to a preceding vehicle.
 13. Thesystem of claim 11, wherein the monitoring information includes handcontact information about contact between the driver and a steeringwheel of the vehicle.
 14. The system of claim 13, wherein the processormodifies control of the vehicle to automatically move based on thevehicular state, the driver state index, and the hand contactinformation.
 15. The system of claim 13, wherein the processor modifiescontrol of the vehicle to automatically move without manual input fromthe driver based on the vehicular state, the driver state index, and thehand contact information.
 16. The system of claim 13, wherein theprocessor determines the head-look of the driver is forward-lookingrelative to the vehicle based on the head movement information, and theprocessor determines there is no hand contact with the steering wheelbased on the hand contact information, the processor controls thevehicle to automatically move forward.
 17. The system of claim 13,wherein the processor determines a plurality of driver states based onthe monitoring information, wherein the plurality of driver statesincludes a first driver state comprising the head movement informationand a second driver state comprising the hand contact information. 18.The system of claim 17, wherein the processor determines a combineddriver state index based on the plurality of driver states, and theprocessor modifies control of the vehicle based on the combined driverstate index.
 19. The system of claim 11, wherein the processordetermines the vehicle is in a stopped state based on the vehicularstate and the driver is attentive based on the driver state index, theprocessor modifies a control status of the one or more vehicle systemsto auto control without a manual input from the driver.
 20. The systemof claim 19, wherein the one or more vehicle systems includes a lowspeed follow system and an automatic cruise control system.